'FeralA' decoder -- free-to-use

New version of the decoder -- V1.2.0K.   I also just shipped to my project partner, so hasn't been super well tested on Windows, but I have verified it.

Please note the two files newcommands-V1.2.0K.txt and using-V1.2.0K.txt.

I know that the decoder is a little (very) clunky, but a lot of the time you might need to try perhaps up to 4-5 times to get the correct decode.  I provided a pattern for finding a close to accurate decode, and remember that if your machine is moderately fast, you can use the --play switch which allows realtime play, and then can modify the commands really quickly.  (I use command line editing, makes it really fast to change.)

Basically, I suggest starting with the least amount of EQ first, and then progressively use more EQ until it sounds right.  The EQ is mostly done in steps, the distributors didn't seem to use in-between values.  So, first start with a 12kHz filter, the 9kHz, then 6kHz, then 4.5kHz and finally 3kHz.   The filter names are --fgh4 for the 12kHz on down to --fgh0 for the 3kHz.  (it is explained a little in the 'using' file and details in the 'newcommands' file.)

There are also more complex sets of filters -- this has been iterative even for me, but I found that ABBA DOES need a fairly complex set of filters -- take a look at the example in the examples file.

Classical recordings tend to be more simple!!!   Just use the --classical switch, and possibly a special --fgh5 switch that does some special EQ that some classical migh need.  They don't mess with the classical stuff as much as POP stuff.

Here is the download directory:

https://www.dropbox.com/sh/1srzzih0qoi1k4l/AAAMNIQ47AzBe1TubxJutJADa?dl=0

If I catch any bugs, I'll fix the decoder fairly quickly.   If you catch any bugs -- tell me, I'll fix quickly.

John

Spectogram difference between the DHNRDS FeralA output producing a copy of 'Crime of the Century' 'School', and the MFSL version.  Note that the MFSL version is apparently properly decoded -- so the comparison is fair.  Also, the signals were normalized, and resolutions/sample rates were all equalized.

The differences in detail and the background noise/fog is pretty obvious for those used to looking at spectrograms.   Of course, measurements like this don't tell the whole story, but the difference while listening shows the DHNRDS version to actually be even more noticeably clean.

Items to look for -- transients being more clean/precise on the DHNRDS version, and the relative lack of background noise.   Not all of the background is 'hiss', but the fuzz around details is a kind of 'fog' that comes from very complex intermodulation effects.

John

@John Dyson

I ran the following:

    da-avx --outgain=-4.5 --info=2 --fa --basic --input=infile.wav --output=outfile.wav

but I got the following error:

    Time Out:

    Required CPU type: WIN64: x86-64bit with AVX2: (Haswell/Zen/Excavator) or greater

    ***FATAL*** open error code: 2, for input file: "infile.wav"

According to Intel, I do have an AVX compatible processor:

What am I doing wrong?

Similarly, I ran:

    da-win --outgain=-4.5 --info=2 --fa --basic --input=infile.wav --output=outfile.wav

but got the following error:

    Time Out:

    Required CPU type: WIN64: x86-64bit with SSE: Silvermont or greater

    ***FATAL*** open error code: 2, for input file: "infile.wav"

39 minutes ago, lucretius said:

@John Dyson

 

I ran the following:

    da-avx --outgain=-4.5 --info=2 --fa --basic --input=infile.wav --output=outfile.wav

 

but I got the following error:

    Time Out:

    Required CPU type: WIN64: x86-64bit with AVX2: (Haswell/Zen/Excavator) or greater

    ***FATAL*** open error code: 2, for input file: "infile.wav"

 

According to Intel, I do have an AVX compatible processor:

 

Product Collection	Intel® Xeon® W Processor
Code Name	Products formerly Skylake
Processor Number	W-2133
Instruction Set Extensions	Intel® SSE4.2, Intel® AVX, Intel® AVX2, Intel® AVX-512
# of AVX-512 FMA Units	2

 

What am I doing wrong?

 

Similarly, I ran:

    da-win --outgain=-4.5 --info=2 --fa --basic --input=infile.wav --output=outfile.wav

 

but got the following error:

    Time Out:

    Required CPU type: WIN64: x86-64bit with SSE: Silvermont or greater

    ***FATAL*** open error code: 2, for input file: "infile.wav"

 

 

 

It seems to be having problems finding the input file 'infile.wav".  

The start-up message always says the kind of CPU that it wants.  If the program gets far enough along to print that message out, then at least it will run on your CPU...

The "Time Out:" message is a leftover vestage of the license manager, which you do not need to use.   The license manager has a timeout time for certain kinds of licenses, but instead of doing the code properly and remove all license manager references, I turned off part of the license manager for when you use the --fa switch, which enables the feralA mode.  I'll remove the rest of the chaffe in the next release.

Basically, I disabled the license manager for the --fa switch as an after thought, but should simply clean it up.

So, the major crux of the problem that you are seeing -- you need a .wav file that you use for input.   Any normal CD wav file should work for testing to see if the program runs for you.  Of course, it will only give you good results if the CD wav file is FeralA (which should be most pop stuff before 1995), and not-normalized (which is a big part of them.)

I should probably supply a .wav file snippet so that at least a quick startup test can be done without selecting a .wav file.  I'll create one in a hour or so, and upload/add it to the repositority.   I'll call it 'testinfile.wav'.

John

2 minutes ago, John Dyson said:

It seems to be having problems finding the input file 'infile.wav".  

 

The start-up message always says the kind of CPU that it wants.  If the program gets far enough along to print that message out, then at least it will run on your CPU...

 

The "Time Out:" message is a leftover vestage of the license manager, which you do not need to use.   The license manager has a timeout time for certain kinds of licenses, but instead of doing the code properly and remove all license manager references, I turned off part of the license manager for when you use the --fa switch, which enables the feralA mode.  I'll remove the rest of the chaffe in the next release.

 

Basically, I disabled the license manager for the --fa switch as an after thought, but should simply clean it up.

 

So, the major crux of the problem that you are seeing -- you need a .wav file that you use for input.   Any normal CD wav file should work for testing to see if the program runs for you.  Of course, it will only give you good results if the CD wav file is FeralA (which should be most pop stuff before 1995), and not-normalized (which is a big part of them.)

 

I should probably supply a .wav file snippet so that at least a quick startup test can be done without selecting a .wav file.  I'll create one in a hour or so, and upload/add it to the repositority.   I'll call it 'testinfile.wav'.

 

John

 

Thank you. I will test with your .wav file.

I'll fix the documentation, but I just checked using this command (similar to the one in the example), but is tuned slightly for the material that I am attaching:

With some luck, that the built-in play feature works, this will play the file (a truncated copy of an ONJ CD RIP):

~/ap/nrs/dabuild/da-avx  --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic  --play

Or, to get an output file (much better tested feature, of course):

~/ap/nrs/dabuild/da-avx  --input=testinfile.wav --output=testoutfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic

Note that the 'testoutfile.wav' file MUST not exist before running the program.  It will not overwrite unless you use the

--overwrite switch before the --output command.  This feature is important because recording engineers work with master tapes and don't want to destroy files by mistake.  (It is a just in case, and paranoid/careful behavior for the software.)

If you don't specify the --fgh2 switch, then the material will be too tinny, it just so happens that the ONJ material likes

the --fgh2 switch.  That switch can be further tweaked, but the results without the --fhh2=xxx tweak will still be okay.

For the higher quality decoding, you can try the --normal switch instead of --basic, or for the super quality (and slow) decodes, you can try --xtra, or even --xpp instead of --basic.

The --tone value isn't very critical, and will be between -12.80 and -12.95, but errors aren't killers unless doing a precision decode.

--info=2 gives the second by second status.  --info=1 gives dots.  --info=0 gives no running status.

John

testinfile.wav

5 hours ago, John Dyson said:

I'll fix the documentation, but I just checked using this command (similar to the one in the example), but is tuned slightly for the material that I am attaching:

 

With some luck, that the built-in play feature works, this will play the file (a truncated copy of an ONJ CD RIP):

 

~/ap/nrs/dabuild/da-avx  --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic  --play

 

Or, to get an output file (much better tested feature, of course):

 

~/ap/nrs/dabuild/da-avx  --input=testinfile.wav --output=testoutfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic

 

Note that the 'testoutfile.wav' file MUST not exist before running the program.  It will not overwrite unless you use the

--overwrite switch before the --output command.  This feature is important because recording engineers work with master tapes and don't want to destroy files by mistake.  (It is a just in case, and paranoid/careful behavior for the software.)

 

If you don't specify the --fgh2 switch, then the material will be too tinny, it just so happens that the ONJ material likes

the --fgh2 switch.  That switch can be further tweaked, but the results without the --fhh2=xxx tweak will still be okay.

 

For the higher quality decoding, you can try the --normal switch instead of --basic, or for the super quality (and slow) decodes, you can try --xtra, or even --xpp instead of --basic.

 

The --tone value isn't very critical, and will be between -12.80 and -12.95, but errors aren't killers unless doing a precision decode.

 

--info=2 gives the second by second status.  --info=1 gives dots.  --info=0 gives no running status.

 

John

 

testinfile.wav 9.25 MB · 1 download

Thanks. I tried the above command line on my own file and it works -- In fact, the old command line I first tried (da-avx --outgain=-4.5 --info=2 --fa --basic --input=testinfile.wav --output=testoutfile.wav) also works -- so I cannot explain why it did not work earlier.  Now, I will have to read up on what the various switches do.

I note that the file is upsampled to 32 bit 88.2K and is a lot bigger.  Is there a reason for this?  What happens if  I resample it to 16bit 44.1K?

Also, I cannot get the following command lines to work:

da-avx  --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic  --play

da-avx --info=2 --fa --basic --input=testinfile.wav --play

Probably something to do with sound drivers?

1 hour ago, lucretius said:

 

Thanks. I tried the above command line on my own file and it works -- In fact, the old command line I first tried (da-avx --outgain=-4.5 --info=2 --fa --basic --input=testinfile.wav --output=testoutfile.wav) also works -- so I cannot explain why it did not work earlier.  Now, I will have to read up on what the various switches do.

 

I note that the file is upsampled to 32 bit 88.2K and is a lot bigger.  Is there a reason for this?  What happens if  I resample it to 16bit 44.1K?

 

Also, I cannot get the following command lines to work:

da-avx  --input=testinfile.wav --info=2 --fa --fgh2 --tone=-12.94 --basic  --play

da-avx --info=2 --fa --basic --input=testinfile.wav --play

 

Probably something to do with sound drivers?

The reason for the upsample is partially internal and partially avoiding lossage.   First, the decoder math doesnt' work at 44.1k when you have audio up to 20kHz -- well, it works but there are some problems.  So, it internally upsamples to 66.15kHz to do the processing, and then upsamples again on output up to 88.2kHz.   There are two reasons for the file to be bigger:  higher sample rate and also floating point.   Floating point files are huge, but are what pros like to use.   If you want to shrink the file a little, but lose very little quality, you can use the' '--intout' switch.  The --intout switch changes the decoder output to 24bits signed instead of floating point.   It is perfectly okay also to resample back down to 44.1k or 48k if you want.  For the highest quality, you might want to keep the file at 24bits, perhaps use flac for data compression without quality loss.  In reality, as long as the levels are adequate (you might want to normalize the output), then converting from the floating point to 16bits is perfectly okay.   Many of my demo versions are 48k/16bits, but instead of normalizing, I am careful to make sure that the peak audio level on the output stays between -3dB and -0.20 or -0.30 dB or so by using the --outgain=xxdB switch.   You really don't want to try to use a 16bit audio file when the signal level is -20dB, so that is why I suggest normalizing before converting to 16bits.

The --play switch, it worries me a little because it is new.  There might be some caveats that I don't know about yet.  There is a file called 'aoplay.exe' that should be in the same directory as the program file 'da-win.exe' or 'da-avx.exe'.   It might work also if 'aoplay.exe' is in the current directory or in the 'HOME' directory.   The problem is that your local audio system might not be set-up the way that I expected -- I have little experience in dealing with the Windows audio system (well, I do, but it is from 15yrs ago!!!)  Basically, the da-avx looks in the same place where you ran it from to also find the 'aoplay.exe' program.  That program is what actually accesses the windows audio system -- it is a separate program because of licensing issues about the library that it uses.

Dont be resistant about asking more questions -- I really want to help!!!

John

38 minutes ago, John Dyson said:

The reason for the upsample is partially internal and partially avoiding lossage.   First, the decoder math doesnt' work at 44.1k when you have audio up to 20kHz -- well, it works but there are some problems.  So, it internally upsamples to 66.15kHz to do the processing, and then upsamples again on output up to 88.2kHz.   There are two reasons for the file to be bigger:  higher sample rate and also floating point.   Floating point files are huge, but are what pros like to use.   If you want to shrink the file a little, but lose very little quality, you can use the' '--intout' switch.  The --intout switch changes the decoder output to 24bits signed instead of floating point.   It is perfectly okay also to resample back down to 44.1k or 48k if you want.  For the highest quality, you might want to keep the file at 24bits, perhaps use flac for data compression without quality loss.  In reality, as long as the levels are adequate (you might want to normalize the output), then converting from the floating point to 16bits is perfectly okay.   Many of my demo versions are 48k/16bits, but instead of normalizing, I am careful to make sure that the peak audio level on the output stays between -3dB and -0.20 or -0.30 dB or so by using the --outgain=xxdB switch.   You really don't want to try to use a 16bit audio file when the signal level is -20dB, so that is why I suggest normalizing before converting to 16bits.

 

The --play switch, it worries me a little because it is new.  There might be some caveats that I don't know about yet.  There is a file called 'aoplay.exe' that should be in the same directory as the program file 'da-win.exe' or 'da-avx.exe'.   It might work also if 'aoplay.exe' is in the current directory or in the 'HOME' directory.   The problem is that your local audio system might not be set-up the way that I expected -- I have little experience in dealing with the Windows audio system (well, I do, but it is from 15yrs ago!!!)  Basically, the da-avx looks in the same place where you ran it from to also find the 'aoplay.exe' program.  That program is what actually accesses the windows audio system -- it is a separate program because of licensing issues about the library that it uses.

 

Dont be resistant about asking more questions -- I really want to help!!!

 

John

 

 

 

 

Thanks for the info. Much appreciated.

I have 'aoplay.exe' in the same directory as 'da-avx.exe', I made that directory the current directory and ran 'da-avx.exe' and also 'aoplay.exe' by itself. 'aoplay.exe' just hangs with no output. 

1 hour ago, lucretius said:

Thanks for the info. Much appreciated.

 

I have 'aoplay.exe' in the same directory as 'da-avx.exe', I made that directory the current directory and ran 'da-avx.exe' and also 'aoplay.exe' by itself. 'aoplay.exe' just hangs with no output. 

Okay, I still have some things to do...   It works on my laptop, but I haven't checked it on many other windows boxes.  I have some windows boxes sent to me that I haven't used yet, and might give it a try.  The --play mode is really nice because it allows quick changes for tweaking the results.   An alternative to the --play switch is if you have sox on your system set-up to do realtime play, but I also found out that the normal windows sox doesn't have an immediate play command that works.

I'll look into the problem again -- because that real-time play makes the tweaking-in of feral material much nicer.   Sometimes, you can just do a partial decode also -- just stop the program after N seconds, and then check that partial output file.  I probably wont' be able to effectively address that level of problem for a few days -- but the truncated decode might be the best short-term work around.

Very often, I have to try two or three times (sometimes more) to tweak-in the results.   Look at the 'using' file that actually describes my initial approach before I have to get 'serious' :-).

The absolute best approach is to use Linux, because it works better for command lines, but the fact that the DHNRDS DA/FA isn't GUI is all my fault and really zero expertise at GUI programming.  (Even if the primitives are given to me -- I cannot visualize a presentation -- missing brain cells!!!)

I will look at it though -- AND THANK YOU FOR TRYING.  I hope that it is still usable for you.

John

6 hours ago, John Dyson said:

Okay, I still have some things to do...   It works on my laptop, but I haven't checked it on many other windows boxes.  I have some windows boxes sent to me that I haven't used yet, and might give it a try.  The --play mode is really nice because it allows quick changes for tweaking the results.   An alternative to the --play switch is if you have sox on your system set-up to do realtime play, but I also found out that the normal windows sox doesn't have an immediate play command that works.

 

I'll look into the problem again -- because that real-time play makes the tweaking-in of feral material much nicer.   Sometimes, you can just do a partial decode also -- just stop the program after N seconds, and then check that partial output file.  I probably wont' be able to effectively address that level of problem for a few days -- but the truncated decode might be the best short-term work around.

 

Very often, I have to try two or three times (sometimes more) to tweak-in the results.   Look at the 'using' file that actually describes my initial approach before I have to get 'serious' :-).

 

The absolute best approach is to use Linux, because it works better for command lines, but the fact that the DHNRDS DA/FA isn't GUI is all my fault and really zero expertise at GUI programming.  (Even if the primitives are given to me -- I cannot visualize a presentation -- missing brain cells!!!)

 

I will look at it though -- AND THANK YOU FOR TRYING.  I hope that it is still usable for you.

 

John

 

Hi John,

Thank you for sharing your project. You are indeed very generous!

Could a how-to video tutorial be a future possibility? 

5 hours ago, dkskl said:

Hi John,

Thank you for sharing your project. You are indeed very generous!

Could a how-to video tutorial be a future possibility? 

I am incredibly shy WRT public presentation, but might figure something out.   Maybe a contributor could add to the project?  It seems like the project is needing to grow in some way for it to grow properly.  I AM thinking about this matter.

It seems like some people really do learn/understand better by a physical presentation, but my mind is just the opposite.

You have come up with a VERY IMPORTANT issue, and I/we (the whole community) need to figure out a better way of explaining the use.

Also, I am feeling more and more -- the DHNRDS needs a GUI.

As you know, lots of stuff is happening right now, but I think that we can come up with good solution for painless training :-).

John

3 hours ago, John Dyson said:

I am incredibly shy WRT public presentation, but might figure something out.   Maybe a contributor could add to the project?  It seems like the project is needing to grow in some way for it to grow properly.  I AM thinking about this matter.

It seems like some people really do learn/understand better by a physical presentation, but my mind is just the opposite.

 

You have come up with a VERY IMPORTANT issue, and I/we (the whole community) need to figure out a better way of explaining the use.

 

Also, I am feeling more and more -- the DHNRDS needs a GUI.

 

As you know, lots of stuff is happening right now, but I think that we can come up with good solution for painless training :-).

 

John

 

Thank you John,

No need for a pro set up with an anchorperson as such imo🙂 Footage of a cursor moving around on a screen and maybe a guiding voice would do fine. Screen shots or slide presentations could also do magic.

For the average Joe like me, -and I guess quite a few others, it would be very helpful. But it is a lot to hope for, I know.

Yes. An intuitive GUI with guidelines would be incredible.

Let's see what the future brings… 🙂

Best wishes,

Soren 

21 hours ago, lucretius said:

I note that the file is upsampled to 32 bit 88.2K and is a lot bigger.  Is there a reason for this?  What happens if  I resample it to 16bit 44.1K?

Hi Lucretius

On a better than average system it is unlikely to sound quite as good.😉

 John supplied me with 2 of the tracks from Crime of the Century in 32/88.2 at my request (curiosity) and much to my surprise, (and John's) they sounded a little better again when played using JRiver 26 than the supplied .flac files,even after conversion to .wav for playing.

The 32/88.2 tracks were delivered as .wav files though . A friend of mine who is also giving John some feedback, actually converted the .flac files to .mp3, and played them through his hotel's Reception area's  mediocre music system, and several guests were very impressed and asked if they were remastered .

Regards

Alex

A minor announcement -- I have observed that there just might be some precision problems in some of the math inside of the DHNRDS DA (and FA mode also.)   The problem is that some of the operations 1) must be damned precise, 2) are separated into lots of little smaller operations that require even greater precision.

For example, normal gain control is calculated by new_signal = old_signal * gain.   Nope -- not what the DHNRDS DA does in high quality modes.  Any error in the gain can manifest directly as distortion.   Even though the math is normally single precision floating point, and even though normal gain control doesn't need any more accuracy than that -- the kind of things happening in the DA decoder are really evil wrt math accuracy.  For example, the actual gain multiplications are NOT signal*gain, but are actually signal*gain^(1/16) power.  That is, the decoder uses gain to the 1/16th power.   Think about this -- instead of a gain of -1.0dB being 0.89125, the actual multiplication is 0.99283024.  Note the loss of two full digits of accuracy!!!   That is a huge accuracy loss.  A lot of gains arent -1.0dB but sometimes -0.1dB -- think about the possible noise in such strange math!??    So, that weird 1/16th power number eventually ends up being the 0.89125 in the physical world, but the loss of precision will cause little biases and excessive dithering in the resulting signal.

I scoured the critical parts where the errors can be magnified as I suggest above.  So -- I moved the most critical math to double precision FP.  You might think -- why didn't I use DP to begin with?   The problem is with the monsterous amount of math being used, especially for the anti-MD and anti-IMD distortion control.   Some of the math reflects directly onto the signal, but other bits of math are time domain filtered -- therefore some of the time-errors might not be as obnoxious (causing distortion and unexpected modulation effects) as they might otherwise be.

This latest experiment is that the attack/release filtering, the post filtering for the Hilbert gain calculations and also the final filter for the gain control are all in double precision now.   Believe it or not, there is a VERY VERY VERY noticeable improvement in clarity.  I regret the inability to share the full decoding examples with everyone, but that is why I am making the decoder available to everyone.  Such an improvement in clarity, which worries me - because THAT IS WHAT I EXPECT.  SO, I might be fooled by my own biases.

A more appropriate matter for the PM groups, but mentioning here that I am making a new, re-decoded 'Crime of the Century' that has been tweaked for lowest distortion along with using the new code.   I know that I have been hearing distortion that I know shouldn't be in the signal, and some of the distortion might be in the original signal.   No matter what, my results are telling me that the mathematics improvement has been beneficial.

* ADD ON -- I just figured out that 'Crime' is recorded with a different matrix than the normal POP albums.  The DHNRDS supports two forms of the stereo matrix (the numbers are still variable), but it has to do with how the channels are scaled.  There is an 'A' style of matrix, but most pop uses the 'B' style of matrix.   Unfortunately, the Supertramp Crime of the century prefers the 'A' style of matrix.  The stereo shifts MUCH MUCH less with the 'A' matrix.  I just found this out -- so it will be 6 Hrs before I can update with the AMAZINGLY improved 'Crime' decode!!!!

The new 'Crime' is almost ready, and I'll make it available to the PM group, along with snippets for everyone else.  I regret the inability to share my examples with everyone, but that is why I am making the decoder available.   We will wait until we are dust in the grave before the distributors will do the right thing and produce high quality recordings.

GIve me until tonight to make the snippets available -- I am still doing final QC on these new decodes of 'Crime of the Century'.

 BY FAR, the latent distortion from math errors is MUCH more prominent than ANY roundoff errors.  The improvement is pretty obvious (SO FAR.)

John

I have the new version of 'Crime' technically finished, just have to create the full demo version.  However, I am providing a snippet and a comparative spectrogram.   Superficially, the MFSL version and the DHNRDS versions almost sound the same.  But, listen carefully -- the MFSL version 'mushes' out more, and also there is an odd tone that warbles around 10kHz on the MFSL version.  Note that there are also vestiges of the tone on the DHNRDS version also, but it is much less audible.

If you notice a bit more of an HF 'sheen' on the MFSL -- that is NOT really music, but is the intermod of all the signal together, and that 'sheen' is part of the 'fuzz' in previous spectrogram examples.  The sheen is typical of true DolbyA results, and is NOT the fault of MFSL, but is a simple PAST  fact of life.

So, generally, the DHNRDS version of this snippet is at least the equal of the MFSL version, but I started with a FeralA CD to produce the result.

I am including the comparative spectrogram (greater background fog/noise on the MFSL version and also the obnoxious 8-9-10kHz signal, probably a modulation product from somehwere.)   The DHNRDS version is relatively more clean -- Imagine how good if we could have started wtih an actual master tape!

Also, to compare fairly, I had to down convert the DHNRDS version to 44.1k/16bits.

BTW -- there are some 'precision' enhancements being readied for the DHNRDS -- it mostly helps in very difficult material.  Supertramp doesn't really exercise the 'precision' issues of previous versions of the decoder -- but again, this is a simple ad-hoc comparison using the FUTURE version of the decoder.   The previously released decoder sounds very similar to the new/work-in-progress decoder on this material.

ddemo-DHNRDS.flac ddemo-MFSL.flac

19 hours ago, John Dyson said:

A minor announcement -- I have observed that there just might be some precision problems in some of the math inside of the DHNRDS DA (and FA mode also.)   The problem is that some of the operations 1) must be damned precise, 2) are separated into lots of little smaller operations that require even greater precision.

For example, normal gain control is calculated by new_signal = old_signal * gain.   Nope -- not what the DHNRDS DA does in high quality modes.  Any error in the gain can manifest directly as distortion.   Even though the math is normally single precision floating point, and even though normal gain control doesn't need any more accuracy than that -- the kind of things happening in the DA decoder are really evil wrt math accuracy.  For example, the actual gain multiplications are NOT signal*gain, but are actually signal*gain^(1/16) power.  That is, the decoder uses gain to the 1/16th power.   Think about this -- instead of a gain of -1.0dB being 0.89125, the actual multiplication is 0.99283024.  Note the loss of two full digits of accuracy!!!   That is a huge accuracy loss.  A lot of gains arent -1.0dB but sometimes -0.1dB -- think about the possible noise in such strange math!??    So, that weird 1/16th power number eventually ends up being the 0.89125 in the physical world, but the loss of precision will cause little biases and excessive dithering in the resulting signal.

 

I scoured the critical parts where the errors can be magnified as I suggest above.  So -- I moved the most critical math to double precision FP.  You might think -- why didn't I use DP to begin with?   The problem is with the monsterous amount of math being used, especially for the anti-MD and anti-IMD distortion control.   Some of the math reflects directly onto the signal, but other bits of math are time domain filtered -- therefore some of the time-errors might not be as obnoxious (causing distortion and unexpected modulation effects) as they might otherwise be.

 

This latest experiment is that the attack/release filtering, the post filtering for the Hilbert gain calculations and also the final filter for the gain control are all in double precision now.   Believe it or not, there is a VERY VERY VERY noticeable improvement in clarity.  I regret the inability to share the full decoding examples with everyone, but that is why I am making the decoder available to everyone.  Such an improvement in clarity, which worries me - because THAT IS WHAT I EXPECT.  SO, I might be fooled by my own biases.

Hi John,

Will you be posting an updated version of your decoder which includes the above enhancements?  And will you be releasing the linux version in addition to the Windows version? Thank you.

Also, is it possible to get a list of switches and values used for those albums you have decoded?  -- especially, ABBA: Gold Greatest Hits (1992) and More ABBA Gold; More ABBA HIts (1993), and of course Supertramp: Crime of the Century.   Thanks again.

3 hours ago, lucretius said:

Hi John,

 

Will you be posting an updated version of your decoder which includes the above enhancements?  And will you be releasing the linux version in addition to the Windows version? Thank you.

 

Also, is it possible to get a list of switches and values used for those albums you have decoded?  -- especially, ABBA: Gold Greatest Hits (1992) and More ABBA Gold; More ABBA HIts (1993), and of course Supertramp: Crime of the Century.   Thanks again.

Yes -- to all of the above.   I am probably down for about 1-2days (need another rest), but I have been exhaustively testing the new changes also.  Unless it is life or death, please give me unti about the late Monday or late Tuesday timeframe.   I work in bursts/then back off....  It is backing off time for a few days.  (I still do testing when resting, just with less intensity.)

The new changes modify the gut-level math, even the DA mode.  The math audible improvement is just barely A/B noticeable on ABBA, but not so much on Supertramp type material.  It is the background mix with foreground on ABBA that makes it horrendous on the math and everything else.  (It doesn't have anything to do with the lyrics in the music ;-), the math issue come with the way the music is put together.)   Supertramp has more transient and impact in the sound, so tends to be easier (counterintuitive), because the anti-MD and anti-IMD math is less affected by precision on big gain changes.   When lots of little things are mixed together, it becomes math-hell.  *Even the original math wasn't all that bad, because the gain is/was time domain filtered in a special way (nonlinear) so that the little gain changes at low levels don't really intermodulate as much as a direct gain applications.   After  a quick review, I doubt that the gain errors were worse than 16bit accuracy, but that was still not great.   The critical calculations should now be at least 4 bits more accurate.  (Again, the effects of gain accuracy are not as bad as the signal accuracy errors, because of the time domain filtering, which was also improved to DP.)*

* The previous/current time domain filtering has little to do with the bulk of the anti-MD/anti-IMD capabilities, even though some of the time domain filtering is used by sections of the anti-MD/anti-IMD capabilities.   The 'meat' of the anti-MD and anti-IMD Hilbert filters were always DP except for the highest speed modes for anti-MD/anti-IMD, where the anti-IMD only high speed mode would be used only for the --normal quality results.   Even then, the results are better than DolbyA HW.

Also, on the decoding parameters, the documentation sometimes changes -- not because of the software, but because it does require skill to choose the best parameters.  Frankly, I am getting better at it, EVEN THOUGH the decoder does support practically everything needed.  There are still just a few too many variables to make it childs-play easy (you would NOT believe the original complexity!!!)  I make mistakes, and any set of parameters that I suggest might be improved by people with better taste/hearing than I have.   I'll definitely 'divulge' the Supertramp and ABBA parameters ( those two are perhaps the most tricky ones) along with the others.   A lot of material, like ONJ, are really robust and big errors are almost inaudible, but ABBA/Supertramp are picky as hell.  I use ABBA for testing because it is so very subtle, but I use Supertramp because it interests audiophiles.

* As I learn the 'needs' for decoding, I am able to offer more and more 'hints' for use.  I havent needed to add built-in capabilities for a very long time.  There are some 'last gasp' tuning parameters where if I didn't supply the correct set of filter frequencies, like an in-between value is needed, then those frequencies can be specified.   There are built-ins for fixed filter frequencies, e.g. 3kHz, 4.5kHz, 6kHz, 9kHz, 12kHz for the start of the rolloff, and then each fiter has cutoff for the rolloff stop also.  The pre-defined filters are usually sufficient, but once in a time, maybe a 7.5kHz filter is needed.  It is very simple to re-purpose the 6kHz filter to 7.5kHz by simple switch, with a frequency spec.  Most often, even if 7.5kHz might be ideal -- using 6kHz is NOT really all that bad.  Even with compromises, the decoded versions are almost always superior to non-decoded (the freq response skews are much worse on undecoded FeralA!!!)

Linux version -- sure -- early on, a few years ago, I had a build procedure for Linux, so it shouldn't be too awful hard to re-enable it.  I would be distributing EXACTLY the same code that I test with, but as you know there is more than just packing up the binary.   Also, the Linux version is MUCH MUCH MUCH nicer to use.  Make sure that you have sox, because the SOX play command works really nicely with DHNRDS DA/FA.   The built-in Linux aplay support also works the last time that I tried, but will double check.  The SOX play command does integrate slightly more nicely though.

Figure on Tuesday at the latest, but if I get a surge of energy will make it available sooner...

John

1 minute ago, John Dyson said:

Yes -- to all of the above.   I am probably down for about 1-2days (need another rest), but I have been exhaustively testing the new changes also.  Unless it is life or death, please give me unti about the late Monday or late Tuesday timeframe.   I work in bursts/then back off....  It is backing off time for a few days.  (I still do testing when resting, just with less intensity.)

Thanks a million!  Please take good care of yourself. Take all the time you need and then some.

2 hours ago, John Dyson said:

The SOX play command does integrate slightly more nicely though.

I already had SoX on my Windows 10 computer (I needed it to correct for the pre-emphasis on CDs). But up to now, I did not create play.exe (copy of sox.exe). So I fixed that. Now when I use command play infile.wav, I get the following error:

    play FAIL sox: Sorry, there is no default audio device configured

However, the default audio device is configured in Windows 10 but SoX is not finding it.  Does anyone know how to make the default sound device available at the command line (in cmd shell)? 

1 hour ago, lucretius said:

 

I already had SoX on my Windows 10 computer (I needed it to correct for the pre-emphasis on CDs). But up to now, I did not create play.exe (copy of sox.exe). So I fixed that. Now when I use command play infile.wav, I get the following error:

 

    play FAIL sox: Sorry, there is no default audio device configured

 

However, the default audio device is configured in Windows 10 but SoX is not finding it.  Does anyone know how to make the default sound device available at the command line (in cmd shell)? 

BTW -- I ran into the same problem on Windows, and that is the reason why I added the somewhat 'hacked in' aoplay.exe, (implements the --play switch).  It doesn't appear to work for everyone -- remember the aoplay.exe file must reside in the same directory as the da-avx.exe, but still doesn't appear to work every time.  JUST FYI -- I don't have an answer for the SOX problem on Windows though...

Off topic about 'play' command stuff, but instead about the decoder.  I found a quality improvement -- pretty subtle.  Remember I was writing about the precision improvement?   Well, the gain control signal is decomposed pretty severely -- this allows more careful incremental gain control and filtering the actual gain control operation - basically squeezing the signal * gain calculation around so that the sidebands are concentrated in less audible places.   Anyway -- this decomposing of the gain control signal is nonlinear time domain.  Even though the gain control signal is carefully shaped to make sure that the slew isn't too fast, and the actual gain control operation is twisted all over in time/frequency space -- the chopped up gain control signal is a nonlinear signal derived from the original, desired gain control signal.  Each decomposed gain control operation is nonlinear but is mathematically combined to produce a normal linear gain control operation.

The problem was that the 'little' gain control operations were nonlinear enough to produce certain sidebands that my method cannot fully cancel or twist into the correct space.  This has been a second order problem, but a problem nonetheless.

To mitigate some of the 'far away' sidebands, I added a very high precision filter to the decomposed gain control signals -- those which are derived from the big one.   Adding this filter to the small gain control segments narrows the sideband creation without having a strong effect on the actual gain control shape.   Of course, ANY manipulation of any gain control signal requires that the signal itself be synchronized to the gain control signal -- very tricky stuff.   All that was done -- no problem, my technology in that area is well proven!!!

Anyway -- bottom line about this change.  The slight excessive sidebands that couldn't be 'twisted' by the anti-MD code are now diminished, without negative audible effects.  The actual operation is a bit more complex than I imply in this note, but the cool thing is that the resulting sound is even more smooth and clean!!!

I work so much better when I don't allow myself to be under pressure.  The bad thing though -- unless I do set goals, I just keep on 'improving', but not coming to conclusion.

As I mentioned before, the new release should be coming about Tue at the latest.  This recent 'improvement' came AFTER the previous email that i sent.  The code/design is being studied ALL of the time for improvements...

John

Maybe interesting info for 'ABBA lovers'.  I think that I found a problem with my previous ABBA decoding.  Yet another parameter that is just a little different than other recordings.   I am including these examples to show the extreme clean-up of the vocal chorus over ANY other version, other than the undecoded ones.   Some of these recordings, in full form, have LOTS of dynamic range.  Most of the more popular ABBA recordings were pretty compressed though.  Even the 'compressed' recordings  have more dynamic range when directly decoded.

I will be including the decoding parameters for the ABBA albums.  I have been able to use exactly the same decoding parameters on every album, and only one album needed so post decoding EQ.  (The bass below 1kHz seems to have been about 0.75dB too weak.)

There are some decoding snippet examples below, using the EXACT parameters that I'll be divulging along with the release...  I started the 'Does your Mother Know' snippet 45 seconds into the song because I wanted to capture the vocal chorus.  Also, included some examples from 'Arrival' -- they have that characteristic 'sandpaper' sound -- every clean example that i have sounds a little like this.  I included 'Bobby' because of the chorus.  'Bobby' on the 'Ring Ring' album might need some narrowing of the stereo image -- it seems to be a normal thing on some POP recordings, but I kept to my promise of everything being the same, except for the 'Visitors' album.

The biggest challenge is SuperTrouper -- this will likely be the only version that any of us has heard that is actually properly decoded.  I think that I have one version that was decoded, but sounds not-so-good.  The other versions have the FeralA sound.  I don't know why SuperTrouper is almost always undecoded, even on vinyl -- unless it sounds very bad on a DolbyA.

Don't expect all of these to sound like the commonly available FeralA versions -- the ABBA sound is actually fairly sane and not severely over compressed.

If you do listen to any of these -- listen for the vocals in each chorus example being much more cleanly separated than usual -- this shows the much greater temporal detail, even when using .mp3 as the examples.

John

06. ABBA - Does Your Mother Know-45to100sec.mp3 02. ABBA - Mamma Mia.mp3 01. ABBA - SOS.mp3 01 - Super Trouper.mp3 08. ABBA - Me And Bobby And Bobby's Brother.mp3 07. ABBA - That's Me.mp3 02. ABBA - Dancing Queen.mp3

Minor delay for 1 more day.  The decoder is working 'perfecto', but I want to look at the real-time play issue for Windows.  That is an 'adjunct', but still important if I can figure out the trouble.

The other benefit of the one day delay is that I'll be able to solidify and verify the decoding formulas for various recordings.   There are some fantastic  results for material like SuperTramp -- I just uploaded some to the review site, but not announced them yet.   Also, there is a minor issue about a 'metallic' sound on 'Even in the Quietest Moments'.  (Delayed the review announcement because of some minor dissatisfaction about the 'Quiet' results.)  I haven't uploaded 'Quiet' yet -- but will do so when I decide on the best possible results.

The previous ABBA results are being improved on further.  ABBA is so strange because their sound isn't real -- it is good, but artificial.  I  am finding that some of their recordings might need a slightly different set of parameters -- and decoding all of ABBA takes quite a while.  However, when I mentioned the results in previous post -- the critical feature about 'decoding' isn't the tonal balance, it is the clarity.   The clarity based upon a minor internal tweak of the decoder is even better.  Even just day before yesterday, I made a slight modification to the decoder -- trying to get the correct mix of minimizing the modulation distortion, while maintaining ALL of the retrievable detail (the results are inversely related -- detail vs. modulation distortion.)

* The problem with detail vs. modulation distortion...  If the gain control is allowed to happen too quickly, one might think that the clarity necessarily improves.  That improvement does happen up to a point, but after that certain speed, then the clarity decreases again.  Maximizing clarity is a lot like optimizing the clarity of taking a photographic picture with a lens.   Up to a certain point, opening up the aperture improves the ultimate clarity, but there are two factors fighting it -- lens quality and depth of field.  Optimizing the clarity of the sound through the decoder has a similar set of opposing forces.  That fast attack slew is very similar to a lens aperture being smaller, therefore the maximum theoretical sharpness decreases, even though the faster attack does superficially improve one form of sharpness.   This is also similar in concept to the Fourier issue of time vs. frequency resolution -- probably come from the same basic math.   This is complicated stuff, but finding the optimum is a matter of experience, not so much a clean mathematical calculation...  With the DHNRDS, I have been able to 'violate' some of the 'rules' by doing things similar to the concept of doing ad-hoc time vs. frequency transforms instead of one big fixed one...   (don't try to understand the details, but the general sense is accurate.) 

I need until Wednesday night, about 30Hrs from this posting time (specified this way because of the confusion about local time.)   I REALLY DO want to give the Windows realtime play a try.  Also, there will be a Linux version, which naturally has realtime play when used in conjunction with SOX (need to document that.)   One more bonus about using Linux -- the decoder can run faster on hyperthreaded machines because there is a working --affinity switch, which directs the decoder to run on the 'correct' threads.   The speed benefit is perhaps 20-30% faster on same machine.

John

Good news and not-so-bad news.   The good news:  I think that I have figured out how to make SOX play commands work on Windows -- this would mitigate the problem and incompatibility with use on Linux.

The not-so-bad news: it might take another day to line up the technical issues, check/understand the licensing issues (what I have to distribute along with the SOX binary), and make sure everything works correctly along with solutions to problems if they do occur.

I didn't like my own anciliary audio play facility for Windows.  Since I try to avoid polluting my mind with the details of the Windows OS, I know it is best to use a program like SOX, because someone else already went through the pain of the obscure and obfuscated API :-).  I think that I figured out what has been the problem with the SOX play command.  Not only that, I have another 'cheat' that will support a 'plan B' for using SOX if the main-line method doesn't work.

As if everyone doesn't already know this -- I have a visceral dislike for the Windows API -- but not really against the Windows kernel.  The kernel is okay, but that isn't what programmers normally need to program -- it is the strange and twisted set of complex APIs and dll mechanism that simply could not come from a sane mind.   SO, since I don't do the programming for money, I try my hardest to avoid pain -- and the exquisitely obfuscated Windows programming API is a very effective delivery mechanism of pain for my mind. (Remember, I used to do OSes -- I enjoy beauty and simplicity, not so much pointy and sharp torture instruments -- I mean Microsoft Windows :-)).

I know that I keep begging off making the most BEAUTIFUL FA decoder possible a day at a time, but this will be worth it.  The decoder *IS* ready, but I wanted to get all of the anciliary ducks in order.  The only thing that might hold off at least the decoder tomorrow is if I get sick*

* Starting about 4-5Hrs ago, I started feeling a very noticeable immune system response...  I sure hope that it is just a hysterical mood, or minor bug of some kind..  No worries -- but I have been too disrupted to think clearly today -- that is HONESTLY the only reason for the delay.  No matter what, if I am not well, I'll make the decoder & docs available, but MAYBE without the sox play command stuff -- until I get better.  Maybe, if it is just hysterics, then stuff should be in order tomorrow.

John