VShell is under export control as a strong encryption product by the U.S. Department of Commerce.

For more information on exporting VShell, see the Encryption Export page.

A new DigiCert certificate was issued to VanDyke Software on August 8, 2024, which affects installers built on or after that date.

There are a couple of ways to address this issue:

View the certificate

1. Right click on the installer and select Properties... from the menu.
2. Select the Digital Signatures tab.
3. Select the item in the Signature list and press the Details button.

Viewing the certificate is usually sufficient to resolve the issue. When the installer is launched again, it should show VanDyke Software, Inc. as the verified publisher.

Ignore the error

If the PKI trusted root authority certificates native to Windows are not current/complete, you may need to either (a) update your trusted root and intermediate certificate authorities within Windows, or (b) ignore the error by choosing Yes when prompted with the Unknown Publisher message.

If you need further assistance installing VShell on Windows, please contact us.

There are three elements that are all essential when considering what type of VShell license purchase will work for your organization:

1. VShell "edition"
2. Server machine count
3. Functionality

All of these elements must be considered together when determining the licensing requirements that apply to your planned deployment of VShell.

VShell adheres to open standards for SSH2, SFTP, FTPS, HTTPS, and WebDav. It is expected that any client that conforms to these standards will be supported by VShell.

If you experience an issue connecting to VShell with a particular client application, please contact support@vandyke.com.

Several things cause key exchange failures. If you are having problems, check the following conditions.

1. What type of host key are you using (DSA, ECDSA, Ed25519, RSA)? Make sure that your client supports the algorithm you selected.
2. Make sure that at least one of the ciphers that your client is using matches at least one of the ciphers enabled in the VShell Control Panel.
3. Make sure that at least one of the MACs that your client is using matches at least one of the MACs enabled in the VShell Control Panel.
4. Make sure that you have Enable compression selected in the VShell Control Panel if your client is requesting compression.

For public-key-only authentication to work with VShell for Windows, you must have the VShell Public Key LSA Authentication Module installed. This module comes with VShell and should have been installed by default with the VShell installation package. You may just need to reboot your machine for the module to take affect.

If rebooting your machine does not correct the problem, try reinstalling VShell and be sure that the Public Key LSA Authentication Module installation option is enabled. After installing VShell with the module, you must reboot your machine.

For more information see the tip, VShell Public-Key Authentication Failure Due to LSA Authentication Module.

To run VShell as a user other than "System", follow these steps:

Note: The user account must be a member of the Administrators groups (according to Microsoft's documentation on loading profiles).

1. Ensure that VShell (the service) is being run as a user account that has the following policy settings:
• Act as part of the operating system
• Logon as a service
• Replace a process-level token
2. Stop any VShell service(s) that are currently running.
3. On the machine running VShell, go to the Windows Control Panel and locate and open Services.
1. Right-click on the VanDyke Software VShell entry and select Properties from the menu.
2. Select the Log On page and click on the This account radio button.
4. Enter the user account that you want to use with the appropriate password.
5. Re-start the VShell service(s).
6. VShell should now be running as the user that you entered.

On Windows

On Linux and macOS

Note: The changes are not applied to any existing client connections that are already established; only new connections can be affected.

If VShell could not authenticate and logged the following error message:

Disconnecting from server: Unable to authenticate using any of the configured authentication methods

there is most likely a UMASK problem.

On Linux, the default UMASK is 0002, which means that files are created by default with the group-writable bit set. This causes public-key authentication with vshelld to fail. To prevent this from happening on newly created files, change the setting on your UMASK to make them not group-writable (set them to 0022, for example). To correct this problem with existing files, change the permissions on your .vshell and publickey folders and any files in your publickey folder so that they are writable only by the owner.

If you receive the following error message:

00002: Could not start program (Scraper.exe /X 80 /Y 24 C:\WINDOWS\system32\CMD.EXE): The user does not have the right to create processes; check for "logon locally" user right

First, ensure that the account that you are trying to connect to VShell with has its settings configured as follows:

• In the VShell Control Panel, select the Access Control category. Ensure that the Logon access right is checked.

• In Windows, ensure that the Log on locally option is selected.

1. Open the Start menu and click on Control Panel.
2. Open System and Security.
3. Open Administrative Tools and select Local Security Policy.
4. In the folder tree, select Local Policies / User Rights Assignments.
5. Make sure that the account in question is listed in the Allow log on locally policy and is not listed in the Deny log on locally policy.

You cannot use VShell-generated key pairs with the OpenSSH SSH-agent because the agent is unable to read private keys generated by vkeygen (unlike public keys, private keys do not have a standardized format). However, VanDyke servers and clients do understand private and public keys generated by OpenSSH applications.

If you are connecting to a Windows 2003 (or newer) server using GSSAPI authentication and cannot establish an interactive shell, the problem may be that you do not have access to the cmd.exe program.

To run GSSAPI, VShell uses the MS SSPI interface, which does not provide a way to do interactive logons and, therefore, does not provide the INTERACTIVE security identifier (SID). Under Windows 2003 server, many of the programs in "windows\system32" grant access via the INTERACTIVE SID; if you do not have the INTERACTIVE SID, you will not be able to run cmd.exe.

To work around this issue, you need to be specifically added to the ACL for cmd.exe or have the NETWORK SID added to the ACL for that file.

Note: This issue also applies to cscript.exe, atrib.exe, cacls.exe and other programs.

Many Windows DOS commands such as "del" and "dir" are built-in commands that are only available within the command shell cmd.exe. Remote execution within VShell for Windows does not include running the client-specified command within a command shell. Therefore, the command shell must be run as part of the remote execution as specified from the client's side. Remote execution of cmd /c <command> should work successfully.

For example:

vsh user@host cmd /c dir - remotely list the directory
vsh user@host cmd /c del <filename> - remotely delete a file

To get VIM working with VShell, you must first install VIM for Microsoft Windows. The installer can be downloaded from the following location:

https://sourceforge.net/projects/vim

Once you have the required version of VIM installed, log on to VShell using your SSH client configured in VT220 mode.

Note: If you are connecting to VShell with terminal type TTY (scrollback mode), VIM will not work.

Scraper.exe is VShell's screen-scraping application used to translate data formatted for Windows systems so that it can be accessed by users running terminal emulators. The screen scraper also reformats user input from terminal emulators so that it can be understood by applications running on Windows systems.

Note: Scraper.exe does not run when the client connected to VShell has requested the use of the scrollback buffer. For more information about the scrollback buffer, see the FAQ, "Why doesn't the scrollback buffer seem to work" below.

Your client may not be configured correctly to use the VShell scrollback feature. You need to set up your client to send a terminal type TTY message when it connects.

Note: When using the scrollback buffer mode, depending on your command shell, it may not be possible to use cursor-based applications like Edit, VI or VIM. Also, you may not be able to use the back arrow to get to the first line of a multi-line command (e.g., if the command is very long and wraps to the next line, you may not be able to cursor back up to the first line).

You may need to experiment with command-line switches to optimize the function of your particular command shell.

To configure SecureCRT to use the VShell scrollback buffer, follow these instructions:

1. Open the SecureCRT Session Options dialog and select the Emulation / Advanced category.
2. Check the Terminal type check box, enter TTY in the corresponding entry box, and click the OK button to save your changes.
   

The following steps describe how to configure vshelld to log its messages to a separate file using the syslog (8) facility.

1. Change the vshelld_config (5) option SyslogFacility to a unique syslog (8) facility that is not currently used by any other applications on the system. For example, the following line will cause vshelld to log to syslog using the "local3" facility: SyslogFacility LOG_LOCAL3.
2. Modify syslog.conf (5) so that messages of the facility you chose in the previous step will be logged to a specific file. For example, the following line will steer any messages to the "local3" facility to the file /var/log/vshelld:

   local3.* /var/log/vshelld

   Log messages from vshelld will now be appended to /var/log/vshelld.



3. Depending on your syslog.conf settings, some or all of your vshelld messages may continue to be appended to other files (such as /var/log/messages). To prevent this, use the "none" syslog level to exclude vshelld messages from being sent to other destinations. For example, including "local3.none" in the following line prevents any vshelld messages (if configured as above) from being appended to /var/log/messages:
   
   

   # Log anything of level info or higher (except local3).
   local3.none;*.info /var/log/messages

For more information on configuring vshelld and syslog, see the vshelld_config (5) and syslog.conf (5) manual pages.

If you do not have the GSSAPI library installed on your server or you have a different version than the one required by your version of VShell, you may be seeing GSSAPI library warnings stating that the library could not be loaded. If you want to stop seeing these warnings, remove GSSAPI from the AuthenticationsAllowed option in the vshelld_config file. Once the parameter is removed, the vshelld server will no longer attempt to load the library and, therefore, won't report the error.

For example, change the configuration option to the following:

AuthenticationsAllowed { publickey, password, keyboard-interactive }

This will stop GSSAPI library warnings from appearing in VShell log files.

The following steps will help you get VNC running over VShell. These steps assume that you are using SecureCRT as your port-forwarding client.

On the VShell/VNC server:

1. In the VNC interface, set the following parameters:
1. Check the Accept Socket Connections check box.
2. Set Display number to 02 (this will configure the server to listen on port 5902).
3. Set the password.
Note: Setting Display number directly to 5902 may cause an error with the client.


2. Create the following Registry entry:
   
   

   AllowLoopBack REG_DWORD "1"

   under the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\ORL\WinVNC3

3. Restart the VNC service.

In your SecureCRT client:

1. Select File / Connect... to open the Session Manager.
2. Click on the New Session button and configure a SSH2 session that connects to the VShell/VNC server.
3. Select the Connections / Port Forwarding category and press the Add button.
4. Create a port forward entry with the following settings:
1. Name: VNC (or whatever name is meaningful to you)
2. Local Port: 5902
3. Remote Port: 5902
4. Clear the Destination host is different for the SSH server checkbox.
5. Click on the OK button until you are back at the Session Manager and then click on the Connect button to initiate your new session.

In your VNCViewer:

1. Enter localhost:02 in the Connection details dialog.

Note: VShell 3.9 and later has addressed this issue by no longer using the Windows Side-by-Side assembly (WinSxS) for dependent libraries. If you are running a version of VShell earlier than 3.9, please upgrade to the latest official release. If you are unable to upgrade, please follow the steps below to resolve the problem.

The following information applies to VShell versions 3.0 through 3.8.

Background information

VShell for Windows requires specific DLL files (Microsoft redistributable libraries). If required DLL files are not already available on a machine when VShell is installed, the VShell for Windows installer stores them in the WinSxS directory in order to ensure that they are available for use when the VShell service starts, or when the VShell Control Panel is launched.

In rare situations, a Microsoft Windows update or non-VanDyke software maintenance action may unexpectedly remove required DLLs or corrupt related manifest files located in the WinSxS system folder.

If these unexpected changes occur while the VShell service is running, the error will not be seen until either the VShell service is restarted, or the machine is rebooted.

It is unknown what application, actions, or circumstances cause the removal of these required WinSxS DLL files and/or manifest files.

When this situation occurs, the VShell service fails to start with the following error message:

Could not start the VanDyke Software VShell SSH2/SFTP service on Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.

Additionally, the VShell Control Panel fails to launch, with the system reporting the following error message:

This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.

Resolution

To correct the WinSxS problem, a system administrator can "repair" the VShell installation. This "repair" operation will not modify your VShell configuration, and can quickly be done using the following steps as a guide:

1. Launch the Windows Control Panel as the user that originally installed VShell on the machine.
2. Launch Add/Remove Programs.
3. Find "VanDyke Software VShell" and press the Change button. If "VanDyke Software VShell" does not appear within the list, then you must re-launch the Add/Remove Programs applet within the user context of the account that was originally used to install the VShell application.
4. When the Welcome screen appears, press the Next button.
5. On the Program Maintenance screen, select Repair and press the Next button.
6. On the Ready To Repair screen, press the Install button to repair your VShell installation. Note that a repair will not modify your current VShell configuration, unless you have installed a custom VShell deployment package which includes VShell configuration information.
7. Once the installer has completed the "repair" operation, you should be able to launch the VShell Control Panel and start the VShell SSH2/SFTP service (as well as the VShell FTPS service, if installed).
   
   Note: The repair operation does not affect core Windows operating system files. It only replaces the DLL files that have been unexpectedly removed — files that the VShell service requires for functional operation.

On the Windows platform, VShell’s FTPS/HTTPS implementation utilizes the SChannel crypto library native to the Windows Operating system.

Although FIPS mode may be enabled in VShell (and active for SSH/SFTP connections)…

…FTPS/HTTPS functionality will not be allowed unless FIPS mode is ALSO ENABLED in Windows.

If FIPS mode is ENABLED in VShell but NOT enabled at the operating system level within Windows, VShell’s FTPS/HTTPS logs will display a warning: Not accepting FTPS (HTTPS) connections because VShell FIPS mode is on. For example:

An inspection of your Windows system’s local security policy will likely reveal that in the Security Options section of your Windows machine’s Local Policies, the System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing option is currently Disabled.

To enable FIPS mode for Windows/SChannel, a Windows system admin must edit the Local Security Policy on the Windows machine where VShell is installed and enable the System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing security option.

To summarize, if FIPS mode has been enabled in VShell via an ADM template…

…and the system level configuration has been made:

Then the warn category message in VShell’s FTPS/HTTPS log file becomes an info category message that reads: VShell FIPS mode is enabled and the Microsoft SChannel setting for FIPS is on.

When using the FileZilla to connect to VShell with the SFTP protocol, you might see the error, "Failed to parse returned path", as shown in the example below:

In VShell 4.4 and later, if FileZilla is the client, and the VShell administrator has restricted access by configuring virtual roots in VShell (yet with the default virtual roots configuration otherwise), then you might expect to see the error above.

The reason is that FileZilla is unable to parse the path presented by VShell in Windows-specific drive format (Name:/). This Windows-specific drive format is the format which VShell presents to file transfer clients by default beginning with VShell 4.4. The default configuration of virtual roots changed as of 4.4, so versions prior to 4.4 are not likely to experience this issue.

To Resolve the Issue from the FileZilla Client:
From FileZilla's Site Manager (with your saved session selected), navigate to the Advanced tab and change the Server type: option to DOS with forward slash separators.

After Server type is changed to DOS with forward-slash separators:

To Resolve the Issue on the VShell Server:
If configuration changes to FileZilla are not possible, you can overcome the parsing issue in VShell by enabling the Start client sessions in the virtual "parent folder" option:

However, the VShell administrator would probably want to configure a subconfiguration so other users connecting to VShell are not affected.

Here is an example subconfiguration that enables the option in VShell:

<?xml version="1.0" encoding="utf-8" ?>
<VShell Version="4.4">
   <UseSingleVirtualRoot Type="dword">1</UseSingleVirtualRoot>
</VShell>