Tag Archives: internet connection

How to Copy Large Files over VPN or Other Unreliable Network Connections

Network Error While Copying

Large file transfer over VPN is a problem for many companies for a few reasons, transfer is unreliable, VPN traffic kills the Internet connection, and it is unproductive. This article will show you how to copy large files over VPN or other unreliable network connections, and some of the best software to deal with this, and best practices to deal with large file transfer, and how to ensure file integrity. These are, in my experience, the best way to do it. You have to evaluate yourself, if they work in your environment, and test a lot.

When copying files over VPN, there are a few problems that need to be addressed:

  • file transfer can be easily interrupted,
  • over saturating the VPN connection,
  • redirecting all the available traffic to the VPN connection
  • ensuring the transferred file is undamaged

Let’s talk a bit about all of these, why they are important, and how they affect the success of your file copy. This will give you, hopefully, a clearer image of the process.

Network Interruption – File Transfer Failed

When we talk about large file transfers, an interruption after a few hours of transfer is not a good thing, and if you take in consideration the time spent, and to only realize you have to do it again because the transfer failed. There are many reasons for the network to interrupt, and even a second is enough to corrupt your file. VPN is prone to network interruptions with large files, because it saturates the Internet bandwidth, and when other VPN clients try to use the VPN tunnel, the file transfer will be interrupted.
The fix for this is some kind of transfer resuming at both the server level and the client level. A few client-server protocols that support file resuming are: SMB, FTP, HTTP, rsync.

The easiest way to implement file transfer resuming in a corporate environment is through SMB and robocopy. Robocopy has a feature to copy in restartable network mode, so if the network goes down, it will automatically resume the transfer from where it left. The robocopy option to copy in restartable mode is /z as with the regular copy command.

robocopy /mir /z  X:\source-folder\ \\RemoteServer\RemoteFolder

Rsync has also a resume option, and it works great, but you have to make sure you use the network filesystem and not a locally mounted network directory. The command to transfer will be like this:

rsync -aP juser@server:/RemoteServer/Directory /Home/Local-Directory

Filezilla has an option to resume files after interruption, but there is a timeout setup by default. Make sure you set the timeout to 0, so that you can recover even after a few hours with no connection between server and client. Note, that this is not a good option if you have many clients, since it will keep open connections indefinitely.

Time Out Config for Resuming

Apache has also an option to allow file resuming, which is enabled by default. The disadvantage with Apache is that, by default it doesn’t support file upload. If you are determined to use Apache for this though, there are some ways to do it, you can start your research here: File Upload plugin for Apache

Over Saturation of the VPN Connection

The VPN connection is there for many users, don’t think that your file is the most important think in the world. If you take all of the bandwidth for your file transfer, other users might not perform daily important tasks.

The over saturation of the VPN connection can be avoided by implementing bandwidth limiting at the software level.

You can do that with SMB by using robocopy, at the client level. The command will look like this:

robocopy /mir /IPG:250  X:\source-folder\ \\RemoteServer\RemoteFolder

The IPG parameter is the one that controls the bandwidth; it’s the acronym for Inter Packet Gap, and with an IPG of 250, the transfer rate on a 100 MBps network is around 12.7 MBps. The lower the IPG, the higher the bandwidth saturation. The transfer rate will be different for different network speeds.

With FTP is going to be at the server level, it’s very simple if you use Filezilla FTP server. Just restrict the bandwidth to a safe limit, (note that there is no restriction in the picture).

FileZilla configuration

Apache uses mod_ratelimit to control the bandwidth of its clients. For more information about that take a look at this page:
http://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html

Rsync can also limit the bandwidth at the client level, and the option will be “–bwlimit”. A command to synchronize two folders with rsync, throttling the bandwidth, would look like this:

rsync –bwlimit=3000 /local/folder user@RemoteHost:/remote/backup/folder/

The 3000 means 3000 kbps. IMPORTANT, if you use

Control VPN Traffic

It looks like a simple decision to allow all the traffic possible through the VPN, most companies will determine that VPN traffic has the highest priority. However, in real life there are many non VPN applications that are run from the Internet, and are critical for a business. Booking a flight ticket, using a hosted web application, getting your emails from a hosted email server, etc… So it might make sense to control the maximum bandwidth used by the VPN, and this is especially needed in environments where large file transfers over VPN are very common. The best way to control this is through firewall policies to limit the bandwidth for the VPN destination. On some devices, like the Fortigate firewalls I am using, this is called traffic shaping.  On other devices might be named differently.

File Integrity Verification

There is a mechanism for checking the file integrity with robocopy and rsync.

The perfect tool to make sure your file is identical with the remote one is to check with md5. On Linux this is a package that comes by default in many distributions, on Windows you can use WinMD5, that you can download it here: WinMD5

MD5 checksum

How to Run Network Cables?

The best time to run your cables for a Computer Network is at the construction/renovation time. At this time it is the easiest to run cables and drill all the passage holes. If the building is already done and it doesn’t need any renovation you can still run your cables.

Fishing and Running Network Cables

For a new building, it is wise to run the cables through designated conduits and to calibrate your conduit so additional cables can be added at a later time if needed. Plan junction boxes at each floor or major node.

For an old building, installing conduits might not be possible. In this case you need to tie your cables in a bunch and secure the bunch to walls, ceiling, etc… You need to use a fish tape to fish your cables in the wall cavity. You might need to use a stud finder to avoid obstructing framing. Adjust your outlet’s position accordingly. Very often the help of a second person is needed when fishing cables, especially when you fish the cables from ceiling to the outlet, behind the wall. In this case on person feeds the fish tape down, or up depending on the case, the other person tries to catch the fish tape with another fish tape or something similar. Once the fish tape is on the position, passed through the drywall hole, attach your cables to the fish tape using duct tape. Run a pull line as well, it will be very useful in the future, in the case you want to add additional cables or replace the defective ones.

Rules for running Network Cables
Make sure before drilling any holes into joists, that you follow the building codes.
Take care of the network cable integrity:

  • do not drop objects on the cables
  • do not step on the cables,
  • do over-bend or over-twist the network cable

When you pull your cable through holes do not pull too hard as this could stretch the cable and modify its electrical qualities. Using a pull-string will provide additional firmness.
Try to avoid electric cables and other electromagnetic sources.

Cable Length and Routes

On an Ethernet type network, typically, we use a star network topology. This means that all of the devices connect into a central point, which is usually a network switch. The network switch receives and distributes all the network packets to network devices.

You need to plan your network so that the longest path from any two network devices is less than 100m, (328ft). If this is not possible plan using additional repeaters or switches. Position your network room, to obtain the least cable lengths.  

How to Build a Computer Network

If you own or manage two or more computers you need to connect your computers in a wired network to share Internet Connection and other services.
Here is a complete How To Build a Network.

Whether you are doing it at home or at your workplace, creating a computer network is not very difficult if you are a handy person and you have basic knowledge of computer networks. We will cover here most of the concepts you need to make your own LAN.
Computer Network
The series of articles will show you how to create a physical network and how to configure it to provide various services to your network users. We will focus on TCP/IP networks as this is the most common networking protocol.

Why Do You Need a Network?

You need a network if you need to share services for two or more computers.
Services that can be shared include: File sharing, sharing a network printer, Internet sharing, email services, Intranet, media broadcasting, etc…
In a home network, it is very common to share a printer and the Internet Connection.
In Business environment, the network becomes more complex and many more services will be needed to accommodate the use of many computers in a single network. Such services are DHCP – for automatic IP configuration, DNS – for name to IP resolution, corporate email, Proxy Server – for increased security on Internet, Intranet server – this is a private web server, VOIP phone system, etc…

What are the Limitations of My Network

You Ethernet cables are limited to 100m between any two active devices. If you need more than 100m between any two active devices such as computer and switch or computer to computer, you need to add a repeater or a hub, a switch will work as well, at each additional 100 meters, (or 333 ft.).
Depending on your materials and equipment, your network can run at 10Mb/s, 100Mb/s or 1Gigabit/s.
The most usual is to use Cat 5 or Cat 5e cables and connectors. A Cat 5 network can provide a speed up to 1000Mb/s.
If you need to connect at longer distances the more suitable solution are
Coaxial cable – up to 500M, speed 10Mb. You need special repeaters or hubs or special network cards.
The fiber optic – 10Km or more, speed up to 100Gb, depending on the equipment. It is the most expensive solution; the price though can be affordable for slower connections. For slower connections, most of the cost will consist on running the cable between the two points.
DSL is the cheap way to connect two remote offices if you have an available copper pair between the two offices.

What Do I Need to Make My own Network?

To make your own Ethernet Network you need the following equipment:
UTP/STP networking cable, RJ-45 connectors, Crimping tool, Punch down tool, Keystone Jacks, Wire stripper / Knife, Network Tester, Patch Panel, Network Switching device. The list includes also a stud finder, drywall saw, measuring tape, mounting plate, fish tape, ladder, and the usual tools such as screwdriver, drilling machine, hammer, etc…
Ample descriptions on the tools and materials are provided by following the links.

Computer Network Planning

The typical Ethernet Network has a star topology. That means that you have a central device, (network switch), that connects all of your network participants. This is important for your planning since you will have to run all of your cables to a central point. Make your measurements and place your central point in such manner that all of your cables are 100m or less. This includes the patch-cord as well.
If you have computers farther than 100m you will need to install a repeater. Alternatively you can use other types of connection, (coax, fiber optic, etc…), see the limitation paragraph.
When you run a cable consider the maximum amount of devices that can be stuffed into that office. It is a lot cheaper to run an extra cable or two per each office location than to run a single cable after the initial installation.
Install the keystones as close as possible to the actual location of the computer. Plan this thoroughly and pick the best location so that the patch-cord is out of the way.
Buy extra cable and extra connectors.
Use common paths for your cables whenever possible. If you run ten cables at once saves a lot of work and time.
Plan your cable route. Use ceiling whenever possible, it is the easiest path. Avoid running the cables near big electromagnetic sources.

Buying Computer Network Tools and Supplies

Do not buy cheap tools and materials. If you do, you will be penalized in different ways:
At the installation time you will get all sorts of problems trying to connect poor materials, cheap cable will be damaged more easily when manipulated, connectors will break easier.
In time the quality of the network will decrease if poor supplies and tools are used.
Overall experience and quality will be very poor if you get too cheap. If buying expensive tools is not justified, (a one time job), try to borrow good tools and don’t buy cheap ones.
Follow the links to learn what to look for when you are buying specific materials.
Now that you planned and bought all the Networking tools and supplies you can proceed to the Running the Cables for a Computer Network.