Tag Archives: email

How to Copy Large Files over VPN or Other Unreliable Network Connections

Network Error While Copying

Large file transfer over VPN is a problem for many companies for a few reasons, transfer is unreliable, VPN traffic kills the Internet connection, and it is unproductive. This article will show you how to copy large files over VPN or other unreliable network connections, and some of the best software to deal with this, and best practices to deal with large file transfer, and how to ensure file integrity. These are, in my experience, the best way to do it. You have to evaluate yourself, if they work in your environment, and test a lot.

When copying files over VPN, there are a few problems that need to be addressed:

  • file transfer can be easily interrupted,
  • over saturating the VPN connection,
  • redirecting all the available traffic to the VPN connection
  • ensuring the transferred file is undamaged

Let’s talk a bit about all of these, why they are important, and how they affect the success of your file copy. This will give you, hopefully, a clearer image of the process.

Network Interruption – File Transfer Failed

When we talk about large file transfers, an interruption after a few hours of transfer is not a good thing, and if you take in consideration the time spent, and to only realize you have to do it again because the transfer failed. There are many reasons for the network to interrupt, and even a second is enough to corrupt your file. VPN is prone to network interruptions with large files, because it saturates the Internet bandwidth, and when other VPN clients try to use the VPN tunnel, the file transfer will be interrupted.
The fix for this is some kind of transfer resuming at both the server level and the client level. A few client-server protocols that support file resuming are: SMB, FTP, HTTP, rsync.

The easiest way to implement file transfer resuming in a corporate environment is through SMB and robocopy. Robocopy has a feature to copy in restartable network mode, so if the network goes down, it will automatically resume the transfer from where it left. The robocopy option to copy in restartable mode is /z as with the regular copy command.

robocopy /mir /z  X:\source-folder\ \\RemoteServer\RemoteFolder

Rsync has also a resume option, and it works great, but you have to make sure you use the network filesystem and not a locally mounted network directory. The command to transfer will be like this:

rsync -aP juser@server:/RemoteServer/Directory /Home/Local-Directory

Filezilla has an option to resume files after interruption, but there is a timeout setup by default. Make sure you set the timeout to 0, so that you can recover even after a few hours with no connection between server and client. Note, that this is not a good option if you have many clients, since it will keep open connections indefinitely.

Time Out Config for Resuming

Apache has also an option to allow file resuming, which is enabled by default. The disadvantage with Apache is that, by default it doesn’t support file upload. If you are determined to use Apache for this though, there are some ways to do it, you can start your research here: File Upload plugin for Apache

Over Saturation of the VPN Connection

The VPN connection is there for many users, don’t think that your file is the most important think in the world. If you take all of the bandwidth for your file transfer, other users might not perform daily important tasks.

The over saturation of the VPN connection can be avoided by implementing bandwidth limiting at the software level.

You can do that with SMB by using robocopy, at the client level. The command will look like this:

robocopy /mir /IPG:250  X:\source-folder\ \\RemoteServer\RemoteFolder

The IPG parameter is the one that controls the bandwidth; it’s the acronym for Inter Packet Gap, and with an IPG of 250, the transfer rate on a 100 MBps network is around 12.7 MBps. The lower the IPG, the higher the bandwidth saturation. The transfer rate will be different for different network speeds.

With FTP is going to be at the server level, it’s very simple if you use Filezilla FTP server. Just restrict the bandwidth to a safe limit, (note that there is no restriction in the picture).

FileZilla configuration

Apache uses mod_ratelimit to control the bandwidth of its clients. For more information about that take a look at this page:
http://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html

Rsync can also limit the bandwidth at the client level, and the option will be “–bwlimit”. A command to synchronize two folders with rsync, throttling the bandwidth, would look like this:

rsync –bwlimit=3000 /local/folder user@RemoteHost:/remote/backup/folder/

The 3000 means 3000 kbps. IMPORTANT, if you use

Control VPN Traffic

It looks like a simple decision to allow all the traffic possible through the VPN, most companies will determine that VPN traffic has the highest priority. However, in real life there are many non VPN applications that are run from the Internet, and are critical for a business. Booking a flight ticket, using a hosted web application, getting your emails from a hosted email server, etc… So it might make sense to control the maximum bandwidth used by the VPN, and this is especially needed in environments where large file transfers over VPN are very common. The best way to control this is through firewall policies to limit the bandwidth for the VPN destination. On some devices, like the Fortigate firewalls I am using, this is called traffic shaping.  On other devices might be named differently.

File Integrity Verification

There is a mechanism for checking the file integrity with robocopy and rsync.

The perfect tool to make sure your file is identical with the remote one is to check with md5. On Linux this is a package that comes by default in many distributions, on Windows you can use WinMD5, that you can download it here: WinMD5

MD5 checksum

How To Find Pinterest Unfollowers – Pinterest Unfollow Tool

pinterest-unfollowersHow To Find Pinterest Unfollowers – Pinterest Unfollow Tool

Removing Pinterest users that do not follow you, or they unfollowed you is not and easy task. Pinterest does not make sorting and finding your unfollowers a simple process. In theory, you shouldn’t worry too much about how many people you follow and they don’t follow back, but in practice, there are some advantages to clean your profile.

You followed a few hundred people, and now your feed is filling up your feed with their pins, but they don’t get your latest, and greatest pinning creations. Or they followed you first, and you returned the favor, but later on they unfollowed you so that they look like authorities. Authorities my a… Let’s remove them from your feed, unfollow them.

Pinterest Unfollowers for IOS Enabled Devices (iPhone, or iPad)

The great news is that if you own an Apple device, such as a tablet, or an iPhone, there is an app that does specifically this. The app is called “Who Unfollow Me For Pinterest”, and you can grab it here: Who Unfollow Me For Pinterest – Yong Peng. I don’t own a MAC IOS device, so I can’t tell you how good the app is, but it’s there. A more famous app seems to have disappeared from the iTunes store, but the app was called Followers on Pinterest.

The Hard Way to Unfollow

Now, what do you do if you don’t have an IOS device? I have seen on the Internet the following suggestion:
Step one: Go to your “Followers” tab and copy/paste in a text document all of your followers.
Step Two: After this, go to your “Following” tab and compare it with your text list. Take one name at  a time from your “Following” tab, and search it in your followers text list. If you find the name, then it means that Pinterest user follows you, if you didn’t find them in the list, just unfollow them right away.

The Easier Way To Unfollow on Pinterest

How to get all followers on PinterestI didn’t even considered doing this, because I figured it would take me days to complete the process, and because I knew it had to be easier.

So, I started the work, and the first thing to was obviously to view page source, like any person with a little HTML knowledge would do. That doesn’t work. The code for the page is served dynamically, through Ajax, so you would only see around 115 users at a time. I think this is somewhat done on purpose, so that unfollowing is very hard. It is in Pinterest’s interest, to maintain an artificial buzz, like everybody follows everybody. Anyway, back to work, we need the entire list for our operation, so I started to search again. Then I realized that if we scroll down to the last item in the HTML page, and we right click the user, and choose inspect element, we also get the HTML of the page, except this time we get all the users in the HTML code.  Just right click on the body element, and choose copy as HTML.

Pinterest-Copy-HTMLAfter that paste the HTML code in a notepad, and you have all your users in a document, save that document as Followers-raw-text.txt.

IMPORTANT: Save the text as UTF8 encoding to preserve the special characters, and to avoid a script freeze when saving as Unicode.

Don’t worry if the text looks gibberish to you, you just need to find the following HTML tag: <h3>. Immediately after the tag you will find the username. Don’t try to save the document as HTML and see it in a browser. The browser will crash while loading that long list.

Pinterest Find UsersAt this point you have a few options to further clean up your list of followers, one way is to use Microsoft Word’s find and replace feature, which does a decent job but is not meant for these kind of operations. Another option is to use an advanced string find/filtering utilities, such as grep, sed, awk, etc… Here is a command I wrote to spit out a plain text list with usernames only:

awk-Pinterest-script-two

 

awk-Pinterest-script-one

I couldn’t “convince” WordPress to  allow me to post the command as a text, so I posted it as a picture. Sorry, you have to write down that manually.

At this point you have a list with followers, and a list with your followings. I heard people trying to use text compare utilities, but they don’t work if you have many followers, or if you have followed only partially boards. I know I did that. Another problem is that the usernames in the following and followers lists, are not in perfect synch chronologically, so any text comparison will be tripped. But if you know they are in the same order, it should be easy enough to use notepad++, or Winmerge, to compare the two lists and get your unfollowers.
I wrote a script in awk, to compare the two lists, and give me only the unfollowers, so I don’t have to compare manually the lists. Here is the command that does that:

awk "FNR==NR{a[$0];next}!($0 in a)" list-of-my-followers.txt list-of-followings.txt > unfollowers.txt

The list in the unfollowers.txt will contain your unfollowers. When you search for them in your Following tab, you will see two types of users, some with a button saying “Follow”, and some with a button “Unfollow”. The ones from your unfollowers.txt are either users whom you followed part of their boards, (the button will say Follow), or users whom you followed all of their boards, (the button will say Unfollow). This will prove a bit difficult to clean, but to make it easier, you might want to follow all of their boards, and maybe they’ll follow you back. If not, you can always clean up next time, if you want.

Future Development

The only thing I have to do now, is to figure out the unfollow link structure, so with a few more adjustments with awk and sed, we can create the unfollow links automatically, so we don’t have to search for the users in page. That’s my next step. If you have an idea about that, drop me a line in the comments, or send me a message with the link below.
In the mean time follow me on Pinterest I will follow back, :).

If you want to ask me a question, or if you want the list of your unfollowers delivered by email, send me a message using this form: Contact Me about the Pinterest Script

UPDATE: If all of these look gibberish to you, you can hire this guy on Fiverr, for five bucks he’ll do it for you. Actually, his gig gave me the idea to make my own script. I didn’t want to pay the five dollars. Anyway, here is the link:  Find Unfollowers on Pinterest