Hunting for "Batch Download" parameters

On-line read version

This short manual is written to be understable even for totally beginners and non-technical users. We will focus on two Net Transport features: "Download all by Net Transport" and "Job > Add batch download ...". These features are common for all download managers, so manual is suitable also for them. On the end of document we will mention a few protection techniques we can encounter and also a few basic sumplemental information.

A few examples of content publishing solutions are presented. All of them are widely used and user can meet them almost every minute during surfing. The images are used as provided content, because there relatively small and, for sure, the most popular. It is possible, that URLs on the images presented here do not quite match with real URLs you will see during your trials at training area. Please, ignore this - it is a detail, because the site is reorganized time to time.

When you read this tutorial off-line, the links to the training area's galleries will not work!. Please, press the left Shift and click to the following link: http://www.ntta.szm.sk/trngarea/. An enter page to example galleries should open, so you can browse to the desired example.

Please take a look also at short "Surfer's basic knowledge" manual, where are collected really basic information about URL, temporary internet files, sumplemental tools, etc.

For those, which not reads manual until there is other ways to explore, I have placed here a short table of examples. Please, try to download all images at once from following pages:

	example 1	Typical gallery with directly linked images Mainly free content pages Easy and straightful download
	example 2	Gallery with images on independed pages Solution usually used by frequently updated pages or pirates Also not so hard for download
	example 3	The same gallery in principle as example 2, but different layout
	example 4	Gallery with images on pop-up'ed windows Solution usually used by serious companies or legal distributors Serious bariers (through JavaScript) for a lot of people, but can be downloaded
	example 5	Heavilly scripted gallery Very rare solution - I have no idea why, because an exelent ideas free of charge are used here Almost impossible to download for ordinary user

By the way, I have heard somewhere that knowledge achieve this harder way, are more resistant than facts to be believed.

Content

[ Language ] :: [ Main Index ] :: [ Site Navigation ] :: [ Tutorials ] :: [ Downloads ] :: [ Contact ]

Picture 1: Typical web image gallery. Note the adress-bar, cursor position and arrised status-bar message
[ Full picture ]

There is an typical situation on screen-shot named as picture 1. You can encounter galeries like this more or less every 5 minutes when you browse over them. No matter if site is commercial, free or belongs to pirate thumbnails post. These screen-shot becomes from Example 1 gallery.

We will continue according to flow-chart at scheme 1. Maybe it looks to be complicated, but believe me, you will learn it during minute.

Scheme 1: How is the gallery builded? So where we have to look for downloading parameters?
[ Full picture ]

First of all, we should place cursor over any thumbnail and see what shows at browser status-bar. If message "Done" or message already displayed does not change, try to go over text. But usually a thumbnail is linked to picture. Solutions when only text is linked are almost never used.

[ Contents ]

Download all by Net Transport

Suppose, there is an URL displayed, which finish with appropriate extension. In our case, as you can see on picture 1, or detailed picture 2, it is ".jpg". Screen-shot becomes from Example 1 gallery. Framed layout, as can be seen at Example 3 (screen-shot) or Example 5 (screen-shot) gallery with directly linked pictures is extremally rare. The most important is URL at status bar, everything else is decor - pretty, but no important.

Picture 2: The most important information focused - collage.
Because there are direct linked images, we can download full gallery via simple right-click choosing "Download all by Net Transport".

Generally, the file should have an extension according to its type. But, mainly images, have sometimes completelly different extension. It is possible, because almost all of major browsers can handle images directly - you can drop a JPG image into browser window and the image will be displayed. One of large company, distibutor and reseller of copyrighted images, displays JPG and PNG files as PHP result. Normally PHP returns HTML document (site). It is an image, but you download pure, transparent PHP script and you can see that Net Transport is unable to get file-size.

Other possibility - rare in present, but still alive at forums - is fake a file content. For example a video, MP3 song or cracked-install-archive, is renamed as TXT file or RAR archive. There is only one reason - just to prevent that someone (administrator) will find this file by automatic search engines and delete it. This is not very popular method now, because of Peer2Peer networks - it is not much more anonymous but for sure safer.

All we have to do is right-click somewhere on the site and choose "Download all by Net Transport" from Microsoft Internet Explorer pop-up menu. See scheme 2, or full sized collage (beware: 215 kBy!) and eventually also scheme 1. Then we can choose the files we wish to download. We can do it via different way, but usually the quickest is Choose selection (i.e. path and type selection), eventually followed by handly correction of automatic choose.

In our case, we click button Choose and on the newly open window we can leave marked all servers (for sure), but we mark only jpg extension. Then click Ok. And click Ok again. Add new download window appears with first file properties. Here we should define Category at least, or check Remark as filename option - it can help us to sort the files with simillar filenames later. And we can expect, that image galleries will have simillar, or even the same filenames.

"Remark as Filename" is priceful when you download a lot of plug-ins or archives, etc. Check this and you usually will get not only filename, but even a short meaningful descprition of their function.

Scheme 2: Five steps to download all wanted content, which is linked directly Click picture to see it larger, or download an archived file, where is a full collage!
Step 1: Choose Download all by Net Transport	Step 2: Press Choose for type based selection	Step 3: Mark desired extensions	Step 4: Correct automatic choice, if necessary	Step 5: Define category and download properties

When we done everything correctly, we should see how the gallery is downloaded.

[ Contents ]

Batch Download parameters

What else we can see at status bar? There can be a scripted URL (e.g. .php?, .asp?, .jsp?) followed by mostly numerical parameters after question mark. This is used mostly by serious companies which demand on its own look and update their content frequently. Individual pictures are than putted into HTML template. We will not shown this example, but individual HTML files for every picture are used at Example 2 and Example 3 on its pure form.

Other possibility is the text as "javascript:(". It is usual that, when JavaScript for opening new window is used, it is also prohibited the right-click on page. Often there is an advert or copyright message at status-bar, sometimes animated. This is applicable, of course, on scripted URL too. Example 4 gallery is build like that.

These two situation you can see at picture 4, which is collage with status-bar message focused.

Picture 4: Thumbnails linked to independed HTML pages with images theirself. Collage of two examples with status-bar messages arissed when cursor is placed over thumbnail.

These JavaScript tricks are meaning-less by my opinion, but supprisingly, they can be a serious barrier for a lot of people (I do not mean a children of computer revolution or enginners, but "marketing" employers / assistents). These solutions are usually used by pirates whos have to pay for domain-name, so they try to earn some money. It is often use by professionals (e.g. photographers), which could be very naive - they just check exclussively "protection" tools and not the "break" articles.

Please believe me, there are really effective solutions how to protect your content. Even for free. But JavaScript used alone is not one of them! Even worse, like my lovely M. says: "I do not want it usually, unless someone clearly manifests, that it is forbidden to me to have it." (Female) Psychology ...

Anyway - according to scheme 1 - please press left Shift and click on any thumbnail. A new window should open. Or directly with picture itself, with new page contains a picture (Example 2 and 3 galleries) or with an error-message, in case of windows opened by javascript (Example 4 gallery). At that last case - you can try it at Example 4 gallery, you should see the text "javascript:(" at the address-bar and page-can-not-be-open message at main window (see picture 5) - just close this window and click on thumbnail again, this time without Shift pressed down. A new window with image inside should open correctly, now.

Picture 5: Microsoft Internet Explorer error-page as result of our try to open new window directly, even though it has to be open through JavaScript. Info text says, that requested page is not reacheable now because of server technical troubles or browser insufficient configuration.

Right-click on displayed picture and choose Properties from pop-up menu. A standard Microsoft Windows properties tab will open and URL of picture can be found at the row of same name. You can select it and copy through right-click and use it later as template at Net Transport. See scheme 3, and/or scheme 1 respectivelly. You can note also file-size at kBy and decide if it is worth to download.

Two more things:

I wrote "any thumbnail". If you want to find-out limit URLs, you should click the first-one and the last-one, of course. Please note, that on big galleries, widespread through a few pages, it is possible, that every page will have its own server-path. Because you cannot define two sequences at Net Transport (and any other download manager, too) - and situation like this is unpredictible - you will have to download every page of gallery separatelly. Please also note, that last one displayed thumbnail, need not to be necessary the last one of serie! See this gallery example (screen-shot).

Right-click can not be allowed. As well as, displaying the status-bar and/or adress-bar, or even menu. Firewall and advert-remover can help you at all of these cases. However, if you have menu accessible, you can find-out the image URL viewing the source code (Microsoft Internet Explorer :: View > Source code). Or you can find a source code at Temporary Internet Files directory - first identfy the image itself and then look for html document containing its name. Looking for a html document containing text "Close window" is not bad idea also, in this concrete case.

So, we have an URL template at clipboard. From Net Transport menu choose Job > Add batch download and paste this URL into editable-box(es). Delete everything from sequence and after from left edit-box (i.e. Front part), sequence we will set later, and delete everything before sequence (including sequence itself) from right edit-box (i.e. Rest part).

Now we are going to set sequence. Simply what is changing. Please see the following table 1 for a few examples (for Net Transport ver. 1.50 and later look this remark):

And finally we can hit Ok and we will see "Add new download" dialog for the first file of setted sequence. Here we should fill-up at least referer site. It is not always necessary - a specially when we are not store files information, however referer site could be required as a part of authorization. After all, it is only three clicks. Index page, or image page itself? - individually, image page is usually more sure. Index page has advantage, that you can quicker check what has gone wrong, if it happened. Other settings (destination category, login, etc.) should be done when it is necessary.

Scheme 3: Five (operative) steps to define a batch download parameters for content on individual and/or independed pages Click picture to see it larger, or download archive for full-size collage with for more details.
Step 1: Open the picture page, right-click and choose Properties	Step 2: Copy picture URL. Repeat it for the last one picture, also, to find-out limits	Step 3: Paste URL to Front / End part, delete accordingly	Step 4: Set the limit values and do not forget about number of digits!	Step 5: Copy and fill-up referer site URL, set destination category / directory, etc. and hit Ok.

After we set the first file parameters and press Ok, the download will strarts immediatelly. Therefore "Start instantly" option is check on by default. If for some reason we need to individualize parameters for some files at serie, we have to uncheck "Start instantly" option and then change the appropriate properties at paused file list. But, this is very rare situation and usually we do not know about this in forward.

Please note, that since Net Transport ver. 1.50, the sequences are marked at "classical" way through string '(*)'! For sure you know, that asterisk means 'all'. Table 1 with examples, will look as follows:

Table 2 - for Net Transport 1.50 and over	URL	Wildcard
Serie to download	(... changing part)	From	To	Digit
pic01.jpg .. pic20.jpg	pic(*).jpg	1	20	2
file1.jpg .. file12.zip	file(*).zip	1	12	1
0541001l.jpg .. 0541175l.jpg	0541(*)l.jpg	1	175	3
02_flA.zip .. 02_flK.zip	02_fl(*).zip	a	k	~ n/a ~

[ Back to Table 1 ]

[ Contents ]

Frequently maded mistakes

• Are you sure, you have gave a correct filename and/or URL? Is there no letter (e.g. "l", "i" or "t" - these are hardly vissible on default sans-serif system font) after counter, which you overloked and delete accidentically? See this example pic001l.jpg, pic002i.jpg, pic001t.jpg, pic002h.jpg.
• Setting which is frequently forgotten is number of digits at wildcard. Its default value - 2 digits - is suitable for 9 of 10 cases, so it comes easy to set limits, e.g. from 1 to 105 and automatically press Enter, left digits unchanged. This case is very easy to recognize. In mentioned example, files 100.png and over will be downloaded and files 01.png till 99.png fail.
• Are you sure you provide a correct authorization? You will need to describe authorization for Net Transport as soon as you have to log-in into page. No matter, that the password is not displayed at adress-bar.
• The images are correctly opened at browser, but they are unreacheable for Net Transport. Are you using firewall or advert-remover (independend or browser-integrated)? Can Net Transport communicate with world-wide-web? Maybe you forgot to set localhost at port 8080.
• Do not switch off option "Monitor > Confirmation" when you have switched on option "System Settings :: Other > Delete job from list after being completed". Both are options of Net Transport. You can fall in deep despair, when you will try to make some order at your default directory.
• Please note, that on the some galleries pic10.jpg follows pic1.jpg. It happens, because of files alphabetical sort. You can see it also on internet. So, the last picture displayed on gallery last page will be pic99.jpg - but, it is quite possible that on the beginning (at gallery first page) you will find pictures pic100.jpg, pic101.jpg, etc. See following screen-shot at picture 7 which becomes from this gallery.

Picture 7: Screen-shot of wrongly sorted gallery. Please note, that pic10 follows immediatelly pic1.
So, when the last thumbnail on last page is e.g. pic99 you should check if there is really no continuation (pic100, pic101, ...) on the gallery begin at first page.
[ Full picture ]

[ Contents ]

"Category" feature

As I mention before, at "Download all by Net Transport" section, we can encounter to problem arrised from downloading enormous files of similar, or the same names from different galleries. When all the files will be saved into same category - in fact, into same directory on hard-disk - Net Transport will add a sequential (ordinal) number to end of file-name. This will protect us from overwrite already existing files, but not help us, when we will try to sort galleries to keep related pictures together.

Maybe it happend to you already, when not, please download all of following 5 galleries into same directory (can be default) and look later into it.

You can see something like on picture 8 , respectivelly into this dir-list. And now imagine, that you want to keep related galleries together. Fine, good starting point is sort the files by time-stamp - situation after this sort is actually shown. So, you can guess you must do handly later. Last number as counter indication is really only symbolical help.

Adding an ordinal number to already existed file-name is exclussively protection mechanism against rewriting file (what can result even with application crush). Author, Mr. Kevin Wang, notices me, that there be a special character added also before this number. Probably instead of pic101.jpg, local file will have name pic10?1.jpg.

Picture 8: Complete chaos when files with simillar names are downloaded into the same directory.
Definitelly not a good idea. See also dir-list below.

Do not blame author - he does not suppose to handle multiple downloads in the way as I just described - i.e. putting everything into one directory. He made categories feature and he programmed it really useful and powerful. And he wrote about this at help.

So for those whose do not reads manuals unless there is some other way to try: here is a short recapitulation how to save our time and speed-up post-processing files.

"Category" feature is at the first look simple database system for storing information about downloaded files. Specially, from where they come from. I personally, have been very pleased from option Net Transport: Tools > Options :: System settings > Delete jobs from list after being completed and I switch it on as soon as I discovered it. I need to know original URL at maybe one promile of files - and for this negligible fraction I switch this option off.

Well, but this is not its only purpose. In fact, "Category" feature is file-management system with full drag&drop, renaming, deleting and databases functions (such as commentary and original URLs) can be kept as bonus to it. No important, if you have "Delete jobs from list after being completed" setted on or off. You can still use it - think about it like another job's parameter.

Please, log-in again at first site of galleries pack. Right-click and choose "Download all by Net Transport". Look eventually for this chapter if you have any doubts.

Picture 9: Defining new category at batch download, so all files are going to be kept together at their individual subdirectory.
Step by step collage from Site 1 of Handling the Chaos section.
[ Full picture ]

At "Add new download" dialog do not automatically click Ok, but look at Destination section, and a specially for three-dots-button right of "Category" window. Click it, so Create new category window will open. Enter some operating system valid name - e.g. press key "1" on numeric pad - and a new subcategory with its own dialog will be created. Now, finish your download.

All gallery is saved to its own directory. It has been created just before downloading, so for sure there is nothing else - well, maybe except resident antivir files. And in this way we can continue with as many galleries as we want to. All of them will have files kept together and their later sort is much more easier.

There is also another possibilities. At first, we can create a new (sub-)directory clicking the three-dots-button right of "Directory" row. However, this is a longer way, because first hard-drive must be scanned before displaying standard Microsoft Windows dialog "Select folder" (or simillar) and then we have to click button to create folder itself and eventually once more to rename it.

Second possibility - sometimes priceless - is uncheck "Start instantly" and over created paused files list to choose "Remark as filename" from right-click menu. Alternative texts or texts which refer to files will be used as filename, so you will get a full description in most of cases. But, I do not think that this one feature - excellent, by the way - can fully replace "Category". You should rather use them together.

The categories folders are normal physical folders on your hard drive. When you want to save an information about files, you should do all file-management within Net Transport. If you copy something on system level, Net Transport will not know about this, because it not scan directories at start-up. Something like this is meaningless at download manager. Be a specially aware about deleting sub-categories! If you delete category, you will also delete a subdirectory on hard drive! So, careful.

Please note, that a Scheme feature has been anounced. It should collect a presets for New download dialog. It is quite possible, that directory, resp. category, will be included there also!

[ Contents ]

Difficulties

Maybe, you have got an illussion that you can download now all series you want to, after all of these examples. A specially, when you figure yourself, how to download Example 5 gallery (by the way, have you try print the page?). Well, it is not quite a truth, because web-masters has a few other possibilities to protect their content even on sign-free sites (no authorization is required and/or necessary). You must understand, that almost all services are commercial now, so web-master has limited traffic size (as I have on this domain), or he is tarrificated on trafic volume base. Therefore, it is not always in web-master interest to do nothing against download manager or offline browser access.

1. Web-master can easily make a serie on independed pages, which cannot be defined. Remember, that only one value at one place can be changed! Other cases and possibilities are just impredictible and only way how to handle it at application is made special chars parser and editor. This is really senseless luxury at download manager. To make sequence like this is a matter of few repeated batch-rename or a quater of hour for programming. Here are three examples of this kind of series:

In this case, you have to use offline browser application, such as Teleport Pro or Offline Explorer. With correct settings - and probably fake itself - it should download this.

2. Web-master can make a serie on independed pages using logic like this:

You can of course define parameters for downloading this serie by Net Transport. But, as you can see, you will get 9995 failed downloads. Now, imagine, that there are no 5 pictures but 25 - your downloading will probably not finish until the life of Sun. Fine, this ad absurdum example, but rename a serie of say 150 pics using random ordinary numbers from interval 000 till 999 is not hard to imagine. You still will get 850 failed downloads.

Again, correctly setted offline browser application can help you.

3. The most common possibility, how to display content which is not actually on our web-space is based on frames. All of advert-banners comes to you this way. It is convenient for content-thief, because traffic will be counted not to him, but to original provider.

Therefore most of small and middle-size companies or providers use .htaccess file protection. In result, all framed references - and you are on framed reference, unless you re-enter to destination page directly - will be re-directed to one of special html sites, mostly 404.html ("File not found" error). This file override Appache web-server configuration and it is really powerful - it can stop e.g. download managers and offline-browsers which can not fake themself as internet-browser. Or it can block Netscape Navigator users ... among others.

Unfortuantelly, I cannot show an example of this feature because I have got no permission for rewritting server rules. You can try at this example, but probably will not work. If you are interesting about this feature, please read this original article of nick-name "Billy" titled "Newbie .htaccess Tutorial: Send those evil hotlinkers to picture hell" at http://www.vnwr.com/resources/tutorials/htaccess.html. There are some links to examples also.

4. Web-master can use a script for creating an URL dynamically. This technique is used e.g. by Yahoo or MSN services. It can not be resented off-line also, but users of their groups knows for sure what am I talking about. If you want to check it in practice, please surf to http://groups.yahoo.com/, respectivelly http://groups.msn.com/ and visit any group which contain pictures.

Pictures of Yahoo groups can be downloaded with Offline Explorer, but it is not an easy task. If you want to download an images at their full-quality sizes, you will need correctly programmed firewall additionally. Or, you can start a slide-show and after it finishes, you can copy images from Temporary Internet Files directory.

MSN groups are much harder to handle and in most cases it is quicker to click through serie and save pictures manually.

[ Contents ]

Opera users' specified information

Opera 7 has builded simple firewall in its code, which can (among others) dissable JavaScript tricks. Its validation is the quickest through F12 key. It call a menu item "File > Quick Preferences". So, right-click and viewing URL is always possible.

A new window - a tab with new window, in fact - can be open with combination Shift + left-click. In the same way as in Microsoft Internet Explorer.

Page source code you can view from pop-up menu - choose "Frame > View Source" - or, quicker, press Alt + F3. Example 3 and 5 galleries use a frames - there are different pages for thumbnails and for pictures. You can check yourself and see that they are really different. So, you should always take an attention on what you are clicking on.

When you right-click an image, you will naturally get a different pop-up menu. If you are still interesting of page source code you should choose "Page > Frame > View Code". However, it is unnecessary, because there is directly "Copy image address" command. Do not use "Copy link address" - it can be a completelly different reference. Example 5 gallery shows this difference.

[ Contents ]

Legal stuff

This short manual has been written to show Net Transport full scale capability.

If anyone wants to translate this, he is welcome. If you need a permission, you have got it in forward: "fair use" for non- profit purposses exclussive with source and credits mentionned. Otherwise, please contact me.

Thanks

I like to thanks everyone from that nameless "crowd" - people those sincere helpless expressed on forums gives me a hints. A specially I like to thank Mr. Kevin Wang for his priceless involvement, constructive comments and cooperation.

Any comments, reflections and/or requests that by your oppinion can improve understability and educational quality are welcome and thanks for them in forward. Please use some meaningful subject at your post.

Copyrights

The brandnames protected by trademarks or registered trademarks appear through all text. I have made a conclussion that I use this names exclussively for informational purpose. Therefore, the appropriate trademark symbols are not accompanied every single reference. They are listed only here. Instead of that, there is always producent named This solution I accepted only for convenient of readers and goods of trademarks and rights legal holder, not even mention of aim to disturbe or break these rights in any way.

Adobe® Photoshop® is registered trademark copyrighted by Adobe® Systems Incorporated, © 1989-2002

Arles Image Web Page Creator is copyrighted by Digital Dutch, © 1998 - 2003

CopyNo.com is copyrighted by melonsoft®, © 2002

HTTP as short-name of Hypertext Transfer Protocol is generic terms and intelectual property of World Wide Web Consortium (W3C®), © 1994 - 2003

Java™ is trademark of Sun Microsystems, Inc.

JPEG is intelectual property of Photographics Expert Group and ISO, © 1982

MetaProducts Offline Explorer™ is trademark copyrighted by MetaProducts Corporation, ©1997 - 2003

Opera is copyrighted Opera Software ASA, ©1985 - 2003

Microsoft®, Windows®, Windows® 2000, Windows® XP, IE as short-name of Internet Explorer®, msn® as official short-name of MSN Web Site (msn "butterfly"™) are registered trademarks or trademarks or copyrights of Microsoft Corp., ©1985 - 2003

Net Transport is copyrighted by Kevin Wang, ©2003

PNG as official short-name of Portable Network Graphics is intelectual property of Portable Network Graphics Development Group, World Wide Web Consortium and ISO, © 1996, 2003

teleport pro™ is trademark copyrighted by Tennyson Maxwell Information Systems, Inc., ©1996 - 1999

Yahoo® groups is part of Yahoo! Inc. services, © 2003

Credits

Pictures are made using November3 action by Paul Reid for Adobe Photoshop

Diaframe (35 mm) is made using Sebastien Piconnier of Aktion Graphik® action titled "35 mm Fram", ©2002 for Adobe Photoshop. Simillar action is part of Auto FX® DreamSuite™ Series One plug-in for Adobe Photoshop 4 plug-in standard compatible bitmap editors, ©2002 Auto FX Software.

Galeries are made using evaluation version of Arles Image Web Page Creator v. 5.1.1, © 1998 - 2003 Digital Dutch.

There is one script used at Example 5 gallery, which is an extended version of clasic roll-over image example, which can be found at every book about JavaScript.
I have found an application called CopyNo.com © 2002 by melonsoft®, which is patent pending and use simillar script. So it seems, that we are at least two on this world, whose got simillar idea independently. Consequently we have both rights to copyright. It happens. Application itself can help you, if you do not know how to modify script I have used or you want to protect enormous count of files. By the way, I handle also a Print command.

[ Contents ]

Marián Stach, Prešov, Slovakia, Central Europe
2003-06-17, 2003-06-27

2004-05-03
Added a remark about changing the sequence definition: front/end part to (*) at URL. Added remark about Scheme feature. Deleted collages (pic. 3 and pic. 5) because I need to spare some space - they are only at archive from now.