kamiwa’s Batch Encoder (KBE)

What is it?

KBE is a small tool I originally had written just for myself to help me optimize my DivX video encoding workflow. The video files I encode here are mainly DVB-S captures. So I needed to cut them, to get rid of unwanted commercials.

So the whole process normally involved:

  • demuxing
  • cutting
  • creation of d2v project
  • creation of AviSynth script
  • adding job to VirtualDubMod and encoding the video
  • encoding audio with BeSweet
  • and finally muxing to DivX.

As all this was very time consuming, I decided to automate the whole process. The only thing I still need to do is setting the cut points in the video. The rest happens automatically.

In March 2007 I implemented DVD support into KBE and in September I added support for GraphFiles created with MS GraphEditor.

As KBE has now been running here for more than one and a half years, I thought, perhaps it is time to throw it at the public, and see what you can do with it.

What it isn’t!

It is no DMF/DVD authoring tool. No support for chapters, subtitles or menus included or planned!

What do you need?

DivX Codec

First of all, you need the DivX Codec installed. The GUI doesn’t check, which version of the codec you got installed. It will accept the PRO version, the Free Community version and any version number. But as it doesn’t check, what you got installed, it will allow you to specify any setting that is valid in the current 6.8 Pro version. If your installed DivX codec doesn’t support the setting, the encode job will fail (or at least look rather crappy!). So it is highly recommended to update your codec to the latest PRO release now!

Other stuff

Apart from the DivX codec KBE is a kind of a HyperGUI to a bunch of tools. Most of them are installed with the installer packet. But apart from those tools that come with the installer there are still three more tools you might be needing:

  • AviSynth: This is a MUST. You need it under all circumstances. I excluded it from the installer, because I didn’t want to tamper with a perhaps already existing installation. So if you haven’t installed it yet, get it here and install it.
  • Java Runtime Engine: You’ll only be needing it, if you want to use ProjectX for demuxing your files. I prefer ProjectX over DGIndex as a demuxer, as it got rid of almost all of my A/V sync issues. If you don’t suffer from them, no need to use Project X. The Java Runtime Engine is available for download here.
  • Cuttermaran: If you need to cut your MPEG-II videos prior to encoding, KBE supports including Cuttermaran into your work flow. Can be downloaded from here.

kamiwa’s Batch Encoder

You can get the installer here. (48.9MB? Are you mad?) Apart from kamiwa’s Batch Encoder itself the installer includes:

  • AviSynthPlugins (AutoCrop, Decomb, DGDecode)
  • BeSweet
  • DGIndex
  • DirectShowLib-2005.dll
  • DivXMux
  • MediaInfo.dll
  • Microsoft .NET Framework 2.0
  • MKVMerge
  • Mp4Box
  • ProjectX
  • VirtualDubMod

This should explain the size of the installer. After the download unpack the archive and launch setup.exe by double-clicking it.

Configuring the Batch Encoder

Important: After the setup, go to the folder where you installed KBE (Normally c:\Program Files\kamiwa’s Batch Encoder). You’ll find a subfolder named tools. In this tool folder you’ll find another sub-directory named VirtualDubMod. Execute VirtualDubMod.exe by double-clicking it. VDM will show a few notifications. Click on OK. This is only necessary on the first start of VDM. If you don’t do it, KBE will hang, the first time it calls VDM.

Setting up the application pathes

Start KBE and click on the Tools tab.

Please check that the application pathes match your settings. If not, adjust them! This should only be necessary, after you installed KBE or if you change your setup.

Setting up the work folders

Click on the Video folders tab

Input folder: This is the folder, where you keep the stuff that you want KBE to process. If you simply want to process single files, uncheck folder mode. If you want to process unprotected DVDs, where each movie consists of several VOBs, check folder mode. You can then create a subfolder in the input folder for each movie and put the VOBs that belong to it in this folder. In other words, if folder mode is checked, KBE will look for subfolders in the input folder instead of looking for individual files. Each found folder is then processed as one movie.

ProjectX folder: The first step in the KBE work flow is to demux your input files. This can either happen by using ProjectX or by using DGIndex (depends on what you later specify in the “Batch processing” tab). This folder is the folder where your demuxed files will end up. If you got more than one physical, it is a wise decision to put this folder on a drive that differs from the Input folder drive. The demux process can then read from one drive and write to the other, instead of switching between read and write mode on one drive. This speeds up things, quite a bit.

Cuttermaran: The next (optional) step in the work flow is cutting the files. If specified, KBE will interrupt the processing of your files after the demux process and start up Cuttermaran to allow you to set cut points for each demuxed file. Just save these cuts as Cuttermaran project files in your demux folder. No need to do the actual cutting. KBE will batch process the Cuttermaran project files, as soon as you closed the Cuttermaran application. The Cuttermaran folder is the folder, where the files, cut by Cuttermaran will end up. Again it is a wise idea (if your HW setup allows it) to have this folder on a drive that differs from the ProjectX folder as this will speed up things a little.

DivX Folder: This is the folder where the DivX encoded video and audio files will end up before they get muxed.

iPod Folder: This is the folder where the iPod encoded video and audio files will end up before they get muxed.

DivX Output: The final destination for your DivX files. If possible put this folder on a drive that differs from the above DivX Folder as this speeds up muxing.

iPod Output: The final destination for your iPod files. If possible put this folder on a drive that differs from the above iPod Folder as this speeds up muxing.

IMPORTANT: Make sure that all folders do physically exist and that each folder is set to a different location. Empty input folders, Empty temporary folders aand Reset VirtualDubMod joblist: These buttons will help you to clean up your system after the encode is finished. Use with care! Make sure that you really don’t need any of the files located in these folders anymore. There’s no way of undoing the delete process!

Configuring the DivX Codec

This screen allows you to configure the way you want to encode your movie.

Profile: Select one of the DivX profiles suitable for the device you later want to play your video on.

Quality Mode: Pick the mode you like. Insane will do the highest compression and thus will be quite slow, fastest will create larger files but will be very quick. If you intend to do an encode, based on a given bitrate, use something above Balanced, as due to the higher compression, it will enable the codec to squeeze more “Quality” into this bitrate.

Quantizer: Up to you really! I prefer optimized.

Encoding modes

KBE supports three different modes: Based on bitrate or bits per pixel, 1 pass constant quality with target quantizer, or the so called optimal mode.

Bitrate mode:

Allows you to specify the number of passes and an average bitrate.

Bits per Pixel

Allows you to specify the number of passes and a value for bits per pixel. Internally KBE calculates this value into a bitrate.

One pass constant quality with target quantizer

Allows you to select a target Quantizer (My preferred way of encoding.)

Optimal mode

This mode will take x % (x is specified by sample size, e.g. 20% percent means: take 20 frames, skip 80, take 20, skip 80, …)  from your video, encode this with the specified quantizer, get the bit rate from the result and reduce this to the given quality percentage. Then do a two pass encode with this calculated bit rate. (Guess you could call this compressibility check!)

Audio Format

Determines which audio format you want to have in your DivX. I implemented Lame MP3, AAC and the ability to keep AC3. MP3 Surround is not supported. Choosing MP3 (Fraunhofer) or MP3 (AudX) will most likely lead to a crash. Nevertheless AC3 audio that has 5.1 will be encoded to MP3 using Dolby Pro Logic II.

Video Format

Cropping: The video will always be auto-cropped. If you don’t like that, use a different tool.

Frame width: You can scale the video width. Max width is determined by the DivX profile you choose. The height will be recalculated respecting the Aspect Ratio.

Aspect Ratio: Can either be square pixel (1:1) or “keep”. With “keep” KB will keep 4:3 or 16:9 AR. AFAIK other AR’s aren’t supported by the codec. So if you got anything else, you should rather use square pixel.

Interlacement: You can either keep it or have it de-interlaced (make progressive). No interlacing of progressive input!

Psycho Visual Enhancement (PVE): If you’re on drugs, use it. I prefer to stay away from it.

Noise Reduction: I’d be glad if we had some snow around this time of year.

Container

As we all know, DMF got its pros and contras. So now you got a choice: Instead of DivX you could use MKV or MP4. AVI is not supported and I doubt, it ever will be.

iPod Settings

Yeah, I felt for you, Apple Lovers. DivX can do iPod! (It can’t! Yes of course it can! Proof it!)

I made it simple for you: Leave the settings as they are. You’re only allowed to play with the width slider. If you run into problems with audio encoding, use FAAC.

Batch settings

Depending on what you check or uncheck here, KBE will process your files.

Demux: If you need to demux your source (if you start a new process on MPEG-II source you will need to!) check it!

Use DGIndex: Will use DGIndex instead of ProjectX for demux. If you use DGIndex, you will loose the ability to interrupt you batch process for manual cutting with Cuttermaran.

Interrupt Batch for manual Cut: Will interrupt the process after demux to allow you to cut your files.

Cut: Will process any Cuttermaran project file, found in the demux folder.

Encode DivX Audio:
Encode DivX Video:
Will encode your files according to the settings you specified under DivX settings.

Encode iPod Audio:
Encode iPod Video:
Will encode your files according to the settings you specified under iPod settings.

Mux DivX: Will mux your encodes into the container specified under DivX settings.

Mux iPod: Will mux your encodes into the container specified under iPod settings.

Start: Will start the encoding Process

Credits

Credits go to the authors of the following software projects:

  • AviSynth
  • AviSynth AutoCrop Plugin
  • BeSweet
  • DGIndex, Decomb, DGDecode
  • DirectShowLib-2005
  • MediaInfo
  • MKVMerge
  • Mp4Box
  • ProjectX
  • VirtualDubMod

kamiwa’s Batch Encoder wouldn’t work without the excellent work they have done. I hope that I included everything in the installer package that needs to go along with the redistribution of their work. If something is missing please let me know, and I shall correct it ASAP.

AttachmentSize
BatchEncoderSetup.rar23 bytes
BatchEncoderSource.rar23 bytes

Comments

Very useful.. Really well done!

By Anonymous (not verified)

This seemed to be the solution to all those open process save - open process save - orgies. But...

Right after "Indexing" just when starting "DivXAudioEncode" "kamiwa's Batch Encoder funktioniert nicht mehr" (crashes) with Vista 64 bit.
Regardless of the DivX settings. Of course all cut files have been correctly created (.ac3 .dv2 .mpa .mpv .mpv.info) ... as far as I can tell.

Is there any way I can help?

By Anonymous (not verified)

FInally, after more than 6,000 reads of this post, a feedback! Thank you!

OK. It seems that the problem is caused by the AudioEncoding step.

You say, you got an AC3 and a MPA audio file.

Here's a check list:

  • What are you encoding to? MP3 or MP4?
  • Should AC3 be passed through or should it be encoded as well?
  • Audio Encoding happens through BeSweet of faad (depending on what you specified). Did they get installed? (Check the BeSweet application folder specified under Tools and make sure that BeSweet.exe and/or faad are there.)
  • Does your Vista user account have enough privileges to execute BeSweet or faad on Vista?
  • Does the Temporary DivX folder, which you specified  under the Folder Settings, exist?
  • Does your user account have enough privileges to write to that folder?
  • Vista 64 bit! I must admit I never tried if BeSweet or faad work under Vista 64 bit at all. And I actually can't! No 64 bit hardware here.
  • If you uncheck AudioEncoding under BatchProcessing and thereby skip the audio encoding, the next step that should happen is Video Encoding. VideoEncoding is done by VirtualDubMod. Does it start up okay? Does it output?

Once more, thanks for your feedback.

By kamiwa

Grüß' Gott Kai,

the temporary and the output folders have full access ("Jeder" -> Vollzugriff), the folder structure is correct in KBE (I prefer a similar demuxed, cut, output structure as you).

Java jre1.6.0_03, AviSynth v2.57, DivX v6.8.2.9 and Cuttermaran v1.69a were manually installed long time before KBE. BeSweet and faac (faad???) were correctly copied by your installer. I had to start VirtualDub Mod manually once, because the disclaimer sometimes prevents batch encodings. Also I gave "VirtualDubMod.jobs" full access ("Jeder" -> Vollzugriff). Only the "AC3 Filter" has problems again on my 64 bit system, so I skip the ac3 files (for now).

For testing purposes I stick with very simple DivX settings in KBE: Video: 1 pass, CBR 800. Audio: mp3 (Lame), CBR 128. Filenames are short and simple, no spaces, no Umlauts.

Here is what happens:

1. I use a TransportStream File from a dbox² recording as input
2. (checked only: Demux, Batch..., Schneiden, DivX Video kodieren, DivX muxen) -> *Start*
3. Java starts ProjectX. Following files in the demuxed folder have been created: ac3, d2v, idd, mpa, mpv, ac3.idd, mpa.idd, _log.txt
4. "ManualCut" Cuttermaran starts up. I manually add with the "open video" the mpv file, mpa and ac3 audio files are added automatically. I delete the ac3 audio (for this test). Define Start I-Frame, End I-Frame, add this cut sequence to the cut list. Save this cpf in the demuxed folder. Close Cuttermaran.
5. "BatchCut" Cuttermaran starts again and is cutting. In the cut folder are now: d2v, mpa, mpv, mpv.info files. Cuttermaran shuts down.
6. "Indexing" appears in KBE
7. "DivXVideoEncode" appears in KBE followed immediately by "kamiwa's Batch Encoder funktioniert nicht mehr"

...didn't test it with activated "DivX Audio kodieren".

Is there some kind of logging option? What happens after "Indexing" exactly? Can I simulate or reproduce the encoding process manually?

By Anonymous (not verified)

Unfortuantely there is no logging option. :-(

But I can tell you that it seems you're doing nothing wrong. You're using the same procedure than I do.

What really should happen after the indexing, is that VirtualDubMod should start up and do the encoding. Does it show up at all? I still have the slight suspicion that it somehow got to do with Vista's security settings. On the other hand why does it allow Java, ProjectX and Cuttermaran to start then?

Hang on! Do the AVS files get created at all?

The immediate next step after indexing should be, that KBE creates an AVS file for every d2v file it finds. During that process, medainfo gets used to determine AspectRatio, FrameSize and so on. Additionally I need to determine the autocrop values. This is done by using autocrop.dll. I set its parameters to write a small log file and then simply open the AVS clip with DirectShow and close it again. Then I parse the log file to get te results.

P:S.: WIr können diese Unterhaltung auch auf Deutsch fortsetzen. Wir sollten nur, wenn wir zu einem Ergbnis kommen, am Ende eine kurze englische Zusammenfassung schreiben.

By kamiwa

Nein, die AviSynth Scripte werden nicht erstellt.

Kannst Du mit einem Microsoft Process Explorer Log etwas anfangen? Da steht relativ genau drin, wann was wo und wie passiert. Es wird dreimal versucht das avs File im "cut" Ordner anzulegen, allerdings immer mit dem Ergebnis "NAME NOT FOUND". Alle anderen gecutteten Dateien landen da brav, ein Rechte Problem kann es dann doch eigentlich nicht sein. ...für weitere Analysen fehlt mir nun allerdings das Coder-Knowhow.
Das Log ist gepackt 6MB groß. Per E-Mail? Was würde vor dem At-Zeichen bei Deiner Domain stehen?

By Anonymous (not verified)

Das ist kein großes Geheimnis! :-) Vor dem @ steht auch kamiwa!

EDIT: Häng mal bitte die .d2v Datei mit dran!

By kamiwa

Hi,
I was unable to follow whether this person was able to run your program in Vista 64bit??? I have Vista 64 & would be happy to give it a try if you think it will work. I'll be happy to give feedback too! :-) Thanks for your interest to make your video tools available online.

By Anonymous (not verified)

Nah, it seems that there is a problem on Vista 64. I'll need to integrate some logging into the app to find out what the problem is. Currently my main suspect is MediaInfo.dll. Not sure it's 64 bit compatible.

Stupid thing is that I can't debug the problem here as I haven't gotten any 64bit hardware to run Vista 64 on.

By kamiwa

I just received an email from the author of the "KBE funktioniert nicht mehr" post. He told me that he managed to get KBE running under Vista 64 Bit version. He only had to exchange the mediainfo.dll from the KBE package against the one found under this link:

MediaInfo_0.7.7.1_CLI_Windows_x64.zip

Thanks, mate! Appreciate it!

By kamiwa

Meanwhile I changed my OS from XP to Vista 64 Ultimate here and found the following two problems

No. 1: MediaInfo.dll is available from SourceForge as a 64bit version. Simply replace the current one with that.

No. 2: KBE modifies the VirtualDubMod file that contains VDM's job queue. As this file resides in the application folder and Vista doesn't like apps to alter files in the app directory, KBE crashes at this point. So what I simply did, I copied VDM to the root folder of my D: drive and altered the VDM application path in KBE accordingly.

Since then it's been working fine for me.

BTW: If any of you guys are into C# and .NET and would like to do some fixes, the source code for KBE is available under:

http://subversion.kamiwa.de/svn/repobrowser.svn?path=/&revision=HEAD&name=kamiwa

 

By kamiwa
Subscribe to Comments for "kamiwa’s Batch Encoder (KBE)"