INDEX

(This is under construction..)

  1. Tools for troubleshooting
  2. Is my training ok?
    1. Semi-continuous models
      I've just run the executable..........
    2. Continuous models
      I've just run the executable..........
  3. Is my decoding setup ok?

TOOLS FOR TROUBLESHOOTING

The two most useful executables used for troubleshooting are

  1. cepview:

    This can be use to view the binary feature file you have generated using the feature computation modules provided with SPHINX. You can display up to N dimensions of the feature vectors by using the flag value 

    -d       N
    The output is in ascii as stdout, and can be piped into any file or program which accepts it.

    In the case of MFCCs, depending on the type of DCT implementation, the values that you will see for the first dimension of each vector might range from 0 to a few hundred. Values in the rest of the dimensions are usually smaller. If the numbers in the file are much bigger, or if you see the sequence "NaN" in the ascii output, it indicates that there is some problem with your audio data. The data could be byte swapped, or you may have computed features for the headers as well, or you may have had digital zeros in the recordings and may have forgotten to add a one-bit dither (or so) while computing the features etc.

  2. printp:

    This executable can be used for printing out the binary model paramter files in ascii format. Depending on which file you are want to look at, one or more of the following flags must be set appropriately: 

    [Switch]     [Default] [Description] 
-tmatfn                The transition matrix parameter file name
-mixwfn                The mixture weight parameter file name
-mixws                 Start id of mixing weight subinterval
-mixwe                 End id of mixing weight subinterval
-gaufn                 A Gaussian parameter file name (either for means or vars)
-gaucntfn              A Gaussian parameter weighted vector file
-regmatcntfn           MLLR regression matrix count file
-moddeffn              The model definition file
-lambdafn              The interpolation weight file
-lambdamin   0         Print int. wt. >= this
-lambdamax   1         Print int. wt. <= this
-norm        yes       Print normalized parameters
-sigfig      4         Number of significant digits in 'e' notation

IS MY TRAINING OK?

If you are training models, even though the training seems to be going through smoothly, keep checking the logfiles being generated for possible problems, and also keep checking the various model architecture and model parameter files being generated. The sections that follow discuss how you can track down problems through these checks before you have wasted a lot of CPU and your own time on the training.

The first thing that you must ensure is that the feature files are being correctly computed. If you are about to train models, then make sure that

IS MY TRAINING OK?

I've just run the executable agg_seg

Structure of the correct log file (the lines in red are explanations) 

$SphinxTrain/bin/agg_seg \
 -segdmpdirs /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 \
 -segdmpfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp \
 -segtype all \
 -ctlfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.fileids \
 -cepdir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//feat \
 -cepext feat \
 -ceplen 13 \
 -agc none \
 -cmn current \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -stride 1

[Switch]    [Default] [Value]
-segdmpdirs           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1
-segdmpfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp
-segidxfn                    
-segtype    st        all    
-cntfn                       
-ddcodeext  xcode     xcode  
-lsnfn                       
-sentdir                     
-sentext                     
-ctlfn                /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.fileids
-mllrctlfn                   
-mllrdir                     
-nskip      0         0      
-runlen     -1        -1     
-moddeffn                    
-ts2cbfn                     
-cb2mllrfn                   
-dictfn                      
-fdictfn                     
-segdir                      
-segext     v8_seg    v8_seg 
-ccodedir                    
-ccodeext   ccode     ccode  
-dcodedir                    
-dcodeext   d2code    d2code 
-pcodedir                    
-pcodeext   p3code    p3code 
-ddcodedir                   
-ddcodeext  xcode     xcode  
-cepdir               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//feat
-cepext     mfc       feat   
-ceplen     13        13     
-agc        max       none   
-cmn        current   current
-varnorm    no        no     
-silcomp    none      none   
-feat                 c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-cachesz    200       200    
-stride     1         1      
INFO: ../main.c(141): No lexical transcripts provided
INFO: ../corpus.c(1270): Will process all remaining utts starting at 0
INFO: ../main.c(245): Will produce FEAT dump
INFO: ../main.c(401): Writing frames to one file
INFO: ../agg_all_seg.c(124): [0]
INFO: ../agg_all_seg.c(124): [1000]
INFO: ../agg_all_seg.c(165): Wrote 1229629 frames to /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp

Warnings

  1. Message 
    

    Explanation







Error messages

    

  1. Message

    areadfloat: path_to_cep/utt.mfc: can't alloc data
ERROR: "../corpus.c", line 1460: MFCC read failed. Retrying after sleep...

    

    Explanation

    
This happens when data are byte-swapped or there are very few frames
in utterance. It also happens when your feature file is physically not
present or is inaccessible/unreadable due to some reason.

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    

















IS MY TRAINING OK?







I've just run the executable kmeans




Structure of the correct log file (the lines in red are explanations)



$SphinxTrain/bin/kmeans_init \
 -gthobj single \
 -stride 1 \
 -ntrial 1 \
 -minratio 0.001 \
 -ndensity 256 \
 -meanfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/means \
 -varfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/variances \
 -reest no \
 -segdmpdirs /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 \
 -segdmpfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp \
 -ceplen 13 \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -agc none \
 -cmn current

[Switch]    [Default]          [Value]
-segdir                               
-segext     v8_seg             v8_seg 
-omoddeffn                            
-dmoddeffn                            
-ts2cbfn                              
-lsnfn                                
-dictfn                               
-fdictfn                              
-cbcntfn                              
-maxcbobs                             
-maxtotobs                            
-featsel                              
-ctlfn                                
-cepext     mfc                mfc    
-cepdir                               
-ceplen     13                 13     
-agc        max                none   
-cmn        current            current
-varnorm    no                 no     
-silcomp    none               none   
-feat       1s_12c_12d_3p_12dd c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-segdmpdirs                    /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1
-segdmpfn                      /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp
-segidxfn                             
-fpcachesz  3000               3000   
-obscachesz 92                 92     
-ndensity                      256    
-ntrial     5                  1      
-minratio   0.01               1.000000e-03
-maxiter    100                100    
-mixwfn                               
-meanfn                        /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/means
-varfn                         /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/variances
-method     rkm                rkm    
-reest      yes                no     
-niter      20                 20     
-gthobj     state              single 
-stride     32                 1      
-runlen                               
-tsoff      0                  0      
-tscnt                                
-tsrngfn                              
-vartiethr  0                  0      
INFO: ../main.c(458): No mdef files.  Assuming 1-class init
INFO: ../main.c(1230): 1-class dump file
INFO: ../main.c(1268): Corpus 0: sz==1229629 frames
INFO: ../main.c(1277): Convergence ratios are abs(cur - prior) / abs(prior)
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 2.085017e-01 ...
INFO: ../kmeans.c(130): km iter [2] 3.839041e-02 ...
INFO: ../kmeans.c(130): km iter [3] 1.496473e-02 ...
INFO: ../kmeans.c(130): km iter [4] 8.121040e-03 ...
INFO: ../kmeans.c(130): km iter [5] 5.256977e-03 ...
INFO: ../kmeans.c(130): km iter [6] 3.857913e-03 ...
INFO: ../kmeans.c(130): km iter [7] 2.907482e-03 ...
INFO: ../kmeans.c(130): km iter [8] 2.227546e-03 ...
INFO: ../kmeans.c(130): km iter [9] 1.753199e-03 ...
INFO: ../kmeans.c(130): km iter [10] 1.408711e-03 ...
INFO: ../kmeans.c(130): km iter [11] 1.177712e-03 ...
INFO: ../kmeans.c(130): km iter [12] 1.001406e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 13 sqerr 3.289491e+05 conv_ratio 8.766461e-04
INFO: ../main.c(561): 	best-so-far sqerr = 3.289491e+05
INFO: ../main.c(211): alloc'ing 112Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 2.146148e-01 ...
INFO: ../kmeans.c(130): km iter [2] 3.472337e-02 ...
INFO: ../kmeans.c(130): km iter [3] 1.313343e-02 ...
INFO: ../kmeans.c(130): km iter [4] 7.271150e-03 ...
INFO: ../kmeans.c(130): km iter [5] 4.842221e-03 ...
INFO: ../kmeans.c(130): km iter [6] 3.540583e-03 ...
INFO: ../kmeans.c(130): km iter [7] 2.704598e-03 ...
INFO: ../kmeans.c(130): km iter [8] 2.159868e-03 ...
INFO: ../kmeans.c(130): km iter [9] 1.765137e-03 ...
INFO: ../kmeans.c(130): km iter [10] 1.490874e-03 ...
INFO: ../kmeans.c(130): km iter [11] 1.267887e-03 ...
INFO: ../kmeans.c(130): km iter [12] 1.080588e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 13 sqerr 1.172793e+06 conv_ratio 9.439682e-04
INFO: ../main.c(561): 	best-so-far sqerr = 1.172793e+06
INFO: ../main.c(211): alloc'ing 14Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 1.958686e-01 ...
INFO: ../kmeans.c(130): km iter [2] 7.782481e-02 ...
INFO: ../kmeans.c(130): km iter [3] 4.728356e-02 ...
INFO: ../kmeans.c(130): km iter [4] 3.346227e-02 ...
INFO: ../kmeans.c(130): km iter [5] 2.553782e-02 ...
INFO: ../kmeans.c(130): km iter [6] 1.942101e-02 ...
INFO: ../kmeans.c(130): km iter [7] 1.523729e-02 ...
INFO: ../kmeans.c(130): km iter [8] 1.252560e-02 ...
INFO: ../kmeans.c(130): km iter [9] 1.063390e-02 ...
INFO: ../kmeans.c(130): km iter [10] 9.017836e-03 ...
INFO: ../kmeans.c(130): km iter [11] 7.604756e-03 ...
INFO: ../kmeans.c(130): km iter [12] 6.461693e-03 ...
INFO: ../kmeans.c(130): km iter [13] 5.621540e-03 ...
INFO: ../kmeans.c(130): km iter [14] 4.818882e-03 ...
INFO: ../kmeans.c(130): km iter [15] 4.225533e-03 ...
INFO: ../kmeans.c(130): km iter [16] 3.757179e-03 ...
INFO: ../kmeans.c(130): km iter [17] 3.428229e-03 ...
INFO: ../kmeans.c(130): km iter [18] 3.172649e-03 ...
INFO: ../kmeans.c(130): km iter [19] 2.947088e-03 ...
INFO: ../kmeans.c(130): km iter [20] 2.603584e-03 ...
INFO: ../kmeans.c(130): km iter [21] 2.363332e-03 ...
INFO: ../kmeans.c(130): km iter [22] 2.202385e-03 ...
INFO: ../kmeans.c(130): km iter [23] 1.996926e-03 ...
INFO: ../kmeans.c(130): km iter [24] 1.828798e-03 ...
INFO: ../kmeans.c(130): km iter [25] 1.665084e-03 ...
INFO: ../kmeans.c(130): km iter [26] 1.508150e-03 ...
INFO: ../kmeans.c(130): km iter [27] 1.367824e-03 ...
INFO: ../kmeans.c(130): km iter [28] 1.258108e-03 ...
INFO: ../kmeans.c(130): km iter [29] 1.151303e-03 ...
INFO: ../kmeans.c(130): km iter [30] 1.046084e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 31 sqerr 6.340286e+05 conv_ratio 9.343168e-04
INFO: ../main.c(561): 	best-so-far sqerr = 6.340286e+05
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 1.379302e-01 ...
INFO: ../kmeans.c(130): km iter [2] 3.473147e-02 ...
INFO: ../kmeans.c(130): km iter [3] 1.589796e-02 ...
INFO: ../kmeans.c(130): km iter [4] 9.246683e-03 ...
INFO: ../kmeans.c(130): km iter [5] 6.071259e-03 ...
INFO: ../kmeans.c(130): km iter [6] 4.333799e-03 ...
INFO: ../kmeans.c(130): km iter [7] 3.330068e-03 ...
INFO: ../kmeans.c(130): km iter [8] 2.629796e-03 ...
INFO: ../kmeans.c(130): km iter [9] 2.166465e-03 ...
INFO: ../kmeans.c(130): km iter [10] 1.799240e-03 ...
INFO: ../kmeans.c(130): km iter [11] 1.517641e-03 ...
INFO: ../kmeans.c(130): km iter [12] 1.314749e-03 ...
INFO: ../kmeans.c(130): km iter [13] 1.144948e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 14 sqerr 4.383058e+05 conv_ratio 9.815118e-04
INFO: ../main.c(561): 	best-so-far sqerr = 4.383058e+05
INFO: ../main.c(843): Initializing variances
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(211): alloc'ing 112Mb obs buf
INFO: ../main.c(211): alloc'ing 14Mb obs buf
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(1323): sqerr [0] == 2.574076e+06
INFO: ../main.c(1327): sqerr = 2.574076e+06 tot 1.604393e+03 rms
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/means [1x4x256 array]
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/variances [1x4x256 array]
INFO: ../main.c(1371): No mixing weight file given; none written
INFO: ../main.c(1525): TOTALS: km 0.026x 1.043e+00 var 0.000x 3.315e+00 em 0.000x 0.000e+00 all 0.026x 1.077e+00




Warnings

    

  1. Message

    INFO: ../main.c(214): alloc'ing 241Mb obs buf
INFO: ../main.c(518): Initializing means using random k-means
INFO: ../main.c(521): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 2.012837e-02 ...
WARNING: "../kmeans.c", line 409: Empty cluster 90
INFO: ../main.c(554):   -> Aborting k-means, bad initialization
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 7.065624e-02 ...
INFO: ../kmeans.c(130): km iter [2] 8.197116e-01 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 3 sqerr 9.556706e+07 conv_ratio
4.972178e-05
INFO: ../main.c(563):   best-so-far sqerr = 9.556706e+07
INFO: ../main.c(214): alloc'ing 30Mb obs buf
INFO: ../main.c(518): Initializing means using random k-means
INFO: ../main.c(521): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 5.450990e-03 ...
WARNING: "../kmeans.c", line 409: Empty cluster 20
WARNING: "../kmeans.c", line 409: Empty cluster 42
WARNING: "../kmeans.c", line 409: Empty cluster 155
INFO: ../main.c(554):   -> Aborting k-means, bad initialization
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 5.045656e-03 ...
WARNING: "../kmeans.c", line 409: Empty cluster 32
WARNING: "../kmeans.c", line 409: Empty cluster 241
INFO: ../main.c(554):   -> Aborting k-means, bad initialization

    

    Explanation

    
This may happen if the features are byte-swapped, or if your feature
file is very small (only a couple of frames or so), or if your features are
bad for some other reason. You may have computed cepstra for the header of
the audio files in your corpus too, so you may have garbage vectors in your
feature files. Garbage vectors can bias the codebooks badly. Make sure that
you do not compute feature vectors corresponding to file-headers.  To see
the cepstral values use the executable "seecep" provided with the SPHINX
package. You can use "cepview" too, but it does not check for byte order
and sometimes prints garbage even if the cepstral files are OK. Byte swap
can be handled through the flag "-mach_endian" in the SPHINX front-end
executable.

    
The presence of digital zeros in your waveform files can also cause this
problem. If a bunch of sample values are zero, and if any of the terms come
out to zero in the computation of the melcep, they get set to a large
negative number (the log is skipped in the front-end code). In this case
the features corresponding to the frames will look like garbage. When there
are many zeros in your samples, the garbage number repeats, and this is a
good indication that the problem is due to the presence of digital zeros.  To
avoid this, it is a good idea to add a one-bit dither to your waveform
before computing the features. The SPHINX front-end does this for you
through a flag called "-dither". The dither is added to the entire signal
(and not just to the regions in which it is zero). However, this should not
be a cause for concern because adding dither to the whole signal is known
to affect recognition performance *positively* in the SPHINX.

    

  2. Message

    

    Explanation

    

  3. Message

    

    Explanation

    







Error messages

    

  1. Message

    

    Explanation

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    
















IS MY TRAINING OK?







I've just run the executable mk_mdef_gen




Structure of the correct log file (the lines in red are explanations)



/usr0/egouvea/SourceForge/tmp/sanity/bin/mk_mdef_gen \
 -phnlstfn /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.phonelist \
 -ocimdef /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef \
 -n_state_pm 5

[Switch]      [Default] [Value]
-phnlstfn               /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.phonelist
-inCImdef                      
-triphnlstfn                   
-inCDmdef                      
-dictfn                        
-fdictfn                       
-lsnfn                         
-n_state_pm   3         5      
-ocountfn                      
-ocimdef                /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef
-oalltphnmdef                  
-ountiedmdef                   
-minocc       1         1      
-maxtriphones 100000    100000 
INFO: main.c(95): Will write CI mdef file /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef
INFO: mk_mdef_gen.c(189): 0 single word triphones in input phone list
INFO: mk_mdef_gen.c(190): 0 word beginning triphones in input phone list
INFO: mk_mdef_gen.c(191): 0 word internal triphones in input phone list
INFO: mk_mdef_gen.c(192): 0 word ending triphones in input phone list
INFO: mk_mdef_gen.c(832): 45 n_base, 0 n_tri
INFO: mk_mdef_gen.c(904): Wrote mdef file /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef




Warnings

    

  1. Message

    

    Explanation







Error messages

    

  1. Message

    

    Explanation

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    















IS MY TRAINING OK?







I've just run the executable mk_flat




Structure of the correct log file (the lines in red are explanations)



$SphinxTrain/bin/mk_flat \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.ci.mdef \
 -topo /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.topology \
 -mixwfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/mixture_weights \
 -tmatfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/transition_matrices \
 -nstream 4 \
 -ndensity 256

[Switch]  [Default] [Value]
-moddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.ci.mdef
-mixwfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/mixture_weights
-topo               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.topology
-tmatfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/transition_matrices
-nstream  4         4      
-ndensity 256       256    
../main.c(46): Reading model definition file /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.ci.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 47 total models defined (47 base, 0 tri)
INFO: ../model_def_io.c(564): 282 total states
INFO: ../model_def_io.c(565): 235 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
../main.c(53): 47 models defined
INFO: ../s3tmat_io.c(152): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/transition_matrices [47x5x6 array]
INFO: ../s3mixw_io.c(212): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/mixture_weights [235x4x256 array]





Warnings

    

  1. Message

    

    Explanation







Error messages

    

  1. Message

    INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
../main.c(53): 47 models defined
ERROR: "../topo_read.c", line 102: EOF encounted while reading version number in /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.topology!?

    

    Explanation

    
The topology file has not been made. Check to see if the script that
makes the topology is in its right place, that it is executable, and
that it is able to write the topology file in the appropriate place.

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    





















IS MY TRAINING OK?







I've just run the executable bw




Structure of the correct log file (the lines in red are explanations)



INFO: main.c(176): Compiled on Nov 23 2000 at 12:11:34
/path/bin/bw \
 -moddeffn /path/model_architecture/training.ci.mdef \
 -ts2cbfn .semi. \
 -mixwfn /path/model_parameters/training.ci_semi_flatinitial/mixture
_weights \
 -mwfloor 1e-08 \
 -tmatfn /path/model_parameters/training.ci_semi_flatinitial/transition_matrices \
 -meanfn /path/model_parameters/training.ci_semi_flatinitial/means \
 -varfn /path/model_parameters/training.ci_semi_flatinitial/variances \
 -dictfn /path/etc/training.dic \
 -fdictfn /path/etc/training.filler \
 -ctlfn /path/etc/training.fileids \
 -part 1 \
 -npart 1 \
 -cepdir /path/feat \
 -cepext feat \
 -lsnfn /path/etc/training.transcription \
 -accumdir /path/bwaccumdir/training_buff_1 \
 -varfloor 0.0001 \
 -topn 4 \                                    
 -abeam 1e-90 \
 -bbeam 1e-40 \
 -agc none \
 -cmn current \
 -meanreest yes \
 -varreest yes -2passvar no \
 -tmatreest yes \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -ceplen 13

[Switch]   [Default] [Value]
-moddeffn            /path/model_architecture/training.ci.mdef
-tmatfn              /path/model_parameters/training.ci_semi_flatinitial/transition_matrices
-mixwfn              /path/model_parameters/training.ci_semi_flatinitial/mixture_weights
-meanfn              /path/model_parameters/training.ci_semi_flatinitial/means
-varfn               /path/model_parameters/training.ci_semi_flatinitial/variances
-mwfloor   0.00001   1.000000e-08
-tpfloor   0.0001    1.000000e-04
-varfloor  0.00001   1.000000e-04              
-topn      4         4
-dictfn              /path/etc/training.dic
-fdictfn             /path/etc/training.filler
-ctlfn               /path/etc/training.fileids
-nskip
-runlen    -1        -1
-part                1
-npart               1
-cepext    mfc       feat
-cepdir              /path/feat
-segext    v8_seg    v8_seg
-segdir
-sentdir
-sentext   sent      sent
-lsnfn               /path/etc/training.transcription
-accumdir            /path/bwaccumdir/training_buff_1
-ceplen    13        13
-agc       max       none
-cmn       current   current
-varnorm   no        no
-silcomp   none      none
-abeam     1e-100    1.000000e-90
-bbeam     1e-100    1.000000e-40         
-varreest  yes       yes
-meanreest yes       yes
-mixwreest yes       yes
-tmatreest yes       yes
-spkrxfrm
-mllrmult  no        no
-mllradd   no        no
-ts2cbfn             .semi.
-feat                c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-timing    yes       yes
-viterbi   no        no
-2passvar  no        no
-sildelfn
-cb2mllrfn
-spthresh  0.0       0.000000e+00
-maxuttlen 0         0
-ckptintv
INFO: main.c(193): Reading /path/model_architecture/training.ci.mdef
INFO: model_def_io.c(598): Model definition info:
INFO: model_def_io.c(599): 47 total models defined (47 base, 0 tri)
INFO: model_def_io.c(600): 282 total states
INFO: model_def_io.c(601): 235 total tied states
INFO: model_def_io.c(602): 235 total tied CI states
INFO: model_def_io.c(603): 47 total tied transition matrices
INFO: model_def_io.c(604): 6 max state/model
INFO: model_def_io.c(605): 20 min state/model
INFO: s3mixw_io.c(127): Read /path/model_parameters/training.ci_semi
_flatinitial/mixture_weights [235x4x256 array]
WARNING: "mod_inv.c", line 363: Model inventory n_density not set; setting to 
value in mixw file, 256.
INFO: s3tmat_io.c(126): Read /path/model_parameters/training.ci_semi
_flatinitial/transition_matrices [47x5x6 array]
INFO: mod_inv.c(291): inserting tprob floor 1.000000e-04 and renormalizing
INFO: s3gau_io.c(163): Read /path/model_parameters/training.ci_semi_
flatinitial/means [1x4x256 array]
INFO: s3gau_io.c(163): Read /path/model_parameters/training.ci_semi_
flatinitial/variances [1x4x256 array]
INFO: gauden.c(178): 1 total mgau
INFO: gauden.c(150): 4 feature streams (|0|=12 |1|=24 |2|=3 |3|=12 )
INFO: gauden.c(189): 256 total densities
INFO: gauden.c(97): min_var=1.000000e-04
INFO: gauden.c(167): compute 4 densities/frame
INFO: main.c(289): Will reestimate mixing weights
INFO: main.c(291): Will reestimate means
INFO: main.c(293): Will reestimate variances
INFO: main.c(295): Will NOT reestimate MLLR multiplicative term
INFO: main.c(297): Will NOT reestimate MLLR additive term
INFO: main.c(305): Will reestimate transition matrices
INFO: main.c(320): Reading main lexicon: /path/etc/training.dic
INFO: lexicon.c(242): 2697 entries added from /path/etc/training.dic
INFO: main.c(331): Reading filler lexicon: /path/etc/training.filler
INFO: lexicon.c(242): 12 entries added from /path/etc/training.filler
INFO: corpus.c(1304): Will process all remaining utts starting at 0
INFO: main.c(534): Reestimation: Baum-Welch
column defns
        <seq>
        <id>
        <n_frame_in>
        <n_frame_del>
        <n_state_shmm>
        <avg_states_alpha>
        <avg_states_beta>
        <avg_states_reest>
        <avg_posterior_prune>
        <frame_log_lik>
        <utt_log_lik>
        ... timing info ...                                           



      
utt> 0 st001a000 1415 0INFO: cvt2triphone.c(210): no multiphones defined, 
no conversion done


   498 295  187 477 2.235357e-41 -1.090004e+01 -1.542355e+04 utt 0.801x 
1.026e upd 0.801x 1.025e fwd 0.130x 1.007e bwd 0.669x 1.030e gau 64.676x 0.998e rsts 0.302x 1.041e rstf 0.004x 1.067e rstu 0.001x 0.765e


utt>  1 st001a001 1548    0   468 274  176 450 1.952369e-41 -9.813935e+00 -1.519197e+04 utt 0.764x 1.000e upd 0.763x 1.000e fwd 0.125x 0.998e bwd 0.636x 1.001e gau 57.130x 1.004e rsts 0.300x 1.010e rstf 0.005x 0.638e rstu
 0.001x 0.760e


utt> 2 st001a002 2025  0 780 372  199 505 2.557036e-41 -1.047046e+01 -2.120267e+04 utt 0.878x 1.000e upd 0.878x 1.000e fwd 0.156x 1.002e bwd 0.720x 1.000e gau 72.144x 0.998e rsts 0.337x 1.005e rstf 0.004x 0.882e rstu
 0.001x 0.769e


utt> 3  st001a003  757    0   234 198  141 362 1.814709e-41 -7.576153e+00 -5.735148e+03 utt 0.583x 0.999e upd 0.581x 1.001e fwd 0.091x 1.000e
 bwd 0.489x 1.000e gau 36.585x 1.016e rsts 0.210x 1.107e rstf 0.003x 1.208e rstu 0.001x 1.060e


utt> 4  st001a004  615 0   132 122  93 240 8.520910e-42 -1.349750e+01 -8.300961e+03 utt 0.398x 1.000e upd 0.398x 0.999e fwd 0.072x 0.986e
bwd 0.325x 1.002e gau 17.075x 0.979e rsts 0.154x 0.997e rstf -0.000x 0.000e rstu 0.002x 0.647e    



***********and so on till the last utt>erance***********







Warnings

    

  1. Message

    WARNING: "gauden.c", line 1387: (mgau= 0, feat= 0, density= 202) never observed
WARNING: "gauden.c", line 1387: (mgau= 0, feat= 0, density= 255) never observed
WARNING: "gauden.c", line 1387: (mgau= 0, feat= 1, density= 9) never observed
WARNING: "gauden.c", line 1387: (mgau= 0, feat= 2, density= 23) never observed
WARNING: "gauden.c", line 1387: (mgau= 0, feat= 2, density= 31) never observed

    

    Explanation

    
The first line above 
means that in the codebook for feature stream 0, the codeword 202 was never observed
 all through the training.
Since you've done a VQ with those very feature vectors, the codeword
*should* have been observed at least once! This problem arises
due to the fact that all vectors in the training set are not used 
for vector quantization. The agg-seg executable selects every n_th vector
from the feature files for vector quantization. This is not a problem
when the feature set size is large. However, if this is done for small
amounts of training data, some codewords seemingly become unrepresentative
of any sample in the training set. This sounds anomalous, but remember that
during training, vectors are not considered in isolation. Instead,
the baum-welch executable identifies *groups* of vectors to
associate with any codeword. That is how not seeing a particular
codeword even once becomes possible.

    
If you get such warnings too many times, you should reduce the
value given to the -stride flag in agg-seg, and redo the vector quantization. 
The stride variable decides the
degree of sampling of the vector space. Every stride_th vector is 
selected for the VQ. When the value given is 1, every vector is used.

    
Smaller amounts of training data result in increased number of warnings.
There warnings may appear even if the stride variable is set to 1, but
the number will be greatly reduced.
The number will further reduce with more and more baum-welch iterations, 
as segmentations become more and more accurate.

    

  2. Message

    WARNING: "../accum.c", line 556: The following seno never occur in the input data
        230 231 232 233 234

    
Explanation

    

If you have run the training in parts, the training data in some parts may
not have examples of the phone or triphone to which these senone-ids
correspond. That is not a problem. A check on the norm logfile at the end of the
current bw iteration can tell you if these (or any of these) senones were seen at
all in any other part of the training or not. If the norm logfile also reports
them as unseen, then it means that your training data did not have any
example of the phone or triphone. Check the model-definition file to find
out which models these senones correspond to. It may be that you did have
examples of the phone or triphone in your training data, but those
utterances either got dropped because they couldn't get force-aligned, or
they got dropped during the bw iteration itself because of the failure of
the backward pass.  Check the bw log files for error messages for failure
of backward pass or the force-align log files to see why they didn't get
force aligned. Very often the problem is just that you didn't have
pronunciations for one or more words (they were not present in the
dictionary) in those utterances. 

    
In any case, if the norm file reports these senones as unseen, and
continues to do so for many iterations (ie, if the utterance is not
recovered as the models get better), then you will end up with zero model
parameters for the particular phone/triphone at the end of your
training. That can harm recognition performance. If the phone is a filler,
then you continue to train and just not use the filler during decoding by
dropping it out from the filler dictionary used during decoding. However,
if it is a phone/triphone that cannot be removed from the decode
dictionaries, or more realistically, *should* have been trained as a
separate acoustic entity, then you must retrain with a training dictionary
that does not have that phone. If, for example, the phone is "ZH", you can
just call it "SH" in the training dictionary and retrain with ZH dropped
out from your phonelist. You will of course have to drop it from the decode
dictionary too.

    

  3. Message

    utt> 27 utterancename  306    0WARNING: "../corpus.c", line 1826: LSN utt id, 
pathname/utterancename, does not match ctl utt id, utterancename

    

    Explanation

    
The warnings can be ignored. The warning above means that for the 27th
utterance called "utterancename", your control file contains the entry
"pathname/utterancename", while the corresponding transcript has only been
labeled by "utterancename". So long as the "utterancename" is the same over
this warning, there is no problem. If it isn't, then your control and
transcript files are misaligned. That needs to be corrected before you
train.  

    

  4. Message 

    WARNING: "../mod_inv.c", line 308: # of tied states in mdef file, 755 != # of mixing weight sets, 2965, in ../hmms/09.cd_chmm/1d/mixw
WARNING: "../mod_inv.c", line 328: Model inventory n_density not set; setting to value in mixw file, 1.

    

    Explanation
You are either using the wrong model definition file for this part of the
training, or the wrong model parameter files. check these out.


  5. Message 
    Explanation





Error messages

    

  1. Message

    INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 47 total models defined (47 base, 0 tri)
INFO: ../model_def_io.c(564): 282 total states
INFO: ../model_def_io.c(565): 235 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
WARN: "../s3io.c", line 225: Unable to open /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//mod
el_parameters/tongues3.ci_semi_flatinitial/mixture_weights for reading; No such
file or directory
FATAL_ERROR: "../main.c", line 930: initialization failed
Fri Dec  1 18:36:45 EST 2000                                        

    

    Explanation

    
This can happen when the flat initialization fails. Flat initialization
can fail for many reasons. Look into the logfile for "makeflat" to find
the reason why it failed.

    

  2. Message

    ERROR: "../backward.c", line 706: alpha(1.882464e-10)
sum of alphas * betas (7.930106e-24) in frame 751
ERROR: "../baum_welch.c", line 225: 028c020f ignored

or 

utt> 3 0004N004 447 0 84ERROR: "../backward.c", line 380: final state not
reached ERROR: "../baum_welch.c", line 245: 00/0004N004 ignored
 utt 0.021x 1.088e upd 0.021x 1.000e fwd 0.021x 1.000e bwd 0.000x 0.000e gau 0.008x 1.257e rsts 0.000x 0.000e rstf 0.000x 0.000e rstu 0.000x 0.000e


    

    Explanation

    
These errors usually occur due to bad initialization
and will disappear after a few iterations. More occassionally some transcripts
may be bad. They can be ignored if there aren't too many of them.

    

  3. Message

    s960207001 3090
WARNING: "../mk_phone_list.c", line 149: Unable to lookup I(2) in the lexicon
WARNING: "../next_utt_states.c", line 58: Unable to produce CI phones for utt
ERROR: "../main.c", line 697: Unable to create utterance HMM; skipping...

    utt 0.001x 1.333e upd 0.000x 0.000e fwd 0.000x
0.000e bwd 0.000x 0.000e gau 0.000x 0.000e rsts 0.000x 0.000e rstf 0.000x
0.000e rstu 0.000x 0.000e

    

    Explanation

    
This means that the word I(2) is not present in the training dictionary.
Check whether your training dict has the same pronunciation markers as
the dictionary you used for force-alignment. If this is not so, your training
is wrong.

    

  4. Message

    INFO: ../main.c(757): Normalizing var
 ERROR: "../gauden.c", line 1389: var (mgau=0, feat=2, density=176,
component=1) < 0

    

    Explanation

    
This happens because we use the following formula to estimate variances:

    
variance = avg(x2) - [avg(x)]2 

    
There are a few weighting terms included (the baum-welch "gamma" weights),
but they are immaterial to this discussion. The *correct* way to estimate
variances is

    
variance = avg[(x - avg(x)]2) 

    
The two formulae are equivalent, of course, but the first one is far more
sensitive to arithmetic precision errors in the computer and can result in
negative variances. The second formula is too expensive to compute (we need
one pass through the data to compute avg(x), and another to compute the
variance). So we use the first one in the sphinx and we therefore get the
errors of the kind we see above, sometimes.

    
The error is not critical (things will continue to work), but may be
indicative of other problems, such as bad initialization, or isolated
clumps of data with almost identical values (i.e. bad data).

    
Another thing that usually points to bad initialization is that you may
have mixture-weight counts that are exactly zero (in the case of
semi-continuous models) or the gaussians may have zero means and variances
(in the case of continuous models) after the first iteration.

    
If you are computing semi-continuous models, check to make sure the initial
means and variances are OK. Also check to see if all the cepstra files are
being read properly.

    

  5. Message

    While training untied models, Baum-Wlech exits with the following message:
FATAL_ERROR: "../ckd_alloc.c", line 149: ckd_calloc_3d failed for caller
at ../mod_inv.c(278) at .. /ckd_alloc.c(150).

    

    Explanation

    
Check the untied model-definition file. This can happen if you are training
semi-continuous models and trying
to train more triphones than your machine's memory can handle. Reduce the
number of triphones by setting the flag "maxdesired" to a lower value (number
of triphones) when you generate the untied mdef file. Memory shortage is
much less likely to happen for continuous models.

    

    

  6. Message

    

    

    Explanation


    


  7. Unexpected stops

    
Message

    INFO: ../corpus.c(1206): Will process 1 utts starting at 0
length 471990272  size 1887961088   /* I added printf to trace feature
length in areadfloat.c  */
areadfloat: /tmp/sls-20000131-000-002.mfc: can't alloc data
column defns
        <seq>
        <id>
        <n_frame_in>
        <n_frame_del>
        <n_state_shmm>
        <avg_states_alpha>
        <avg_states_beta>
        <avg_states_reest>
        <avg_posterior_prune>
        <frame_log_lik>
        <utt_log_lik>
        ... timing info ...
utt>     0      sls-20000131-000-002ERROR: "../corpus.c", line 1477:
MFCC read failed.  Retrying after sleep...

    

    Explanation
The feature files are byte-swapped.

    
Message

    

    Explanation




Back to index











IS MY TRAINING OK?







I've just run the executable norm




Structure of the correct log file (the lines in red are explanations)



$SphinxTrain/bin/norm \
 -accumdir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_3 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_4 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_5 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_6 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_7 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_8 \
 -mixwfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/mixture_weights \
 -tmatfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/transition_matrices \
 -meanfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/means \
 -varfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/variances \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -ceplen 13

[Switch]   [Default] [Value]
-accumdir            /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_3 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_4 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_5 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_6 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_7 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_8
-oaccumdir                  
-tmatfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/transition_matrices
-mixwfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/mixture_weights
-meanfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/means
-varfn               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/variances
-regmatfn                   
-dcountfn                   
-inmixwfn                   
-inmeanfn                   
-invarfn                    
-feat                c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-ceplen              13     
INFO: ../feat.c(191): Using features c/1..L-1/,d2/1..L-1/d4/1..L-1/,c/0/d2/0/d1(d2/0/),d1(d2/1..L-1/)
INFO: ../main.c(499): Reading and accumulating counts from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1
INFO: ../s3mixw_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/mixw_counts [235x4x256 array]
INFO: ../s3tmat_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tmat_counts [47x5x6 array]
INFO: ../s3gau_io.c(343): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/gauden_counts with means with vars (2pass) [1x4x256 vector arrays]

INFO: ../main.c(499): Reading and accumulating counts from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2
INFO: ../s3mixw_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2/mixw_counts [235x4x256 array]
INFO: ../s3tmat_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2/tmat_counts [47x5x6 array]
INFO: ../s3gau_io.c(343): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2/gauden_counts with means with vars (2pass) [1x4x256 vector arrays]

*************and so on till all buffers are read**********************

INFO: ../main.c(743): Normalizing mean for n_mgau= 1, n_stream= 4, n_density= 256
INFO: ../main.c(757): Normalizing var
INFO: ../s3mixw_io.c(212): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/mixture_weights [235x4x256 array]
INFO: ../s3tmat_io.c(152): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/transition_matrices [47x5x6 array]
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/means [1x4x256 array]
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/variances [1x4x256 array]
Sat Dec  2 11:06:57 EST 2000
Current Overall Likelihood Per Frame = 2.76028
Convergence Ratio = 0.047867




Warnings

    

  1. Message

    WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 210) never observ
ed
WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 226) never observ
ed
WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 245) never observ
ed
WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 250) never observ
ed                                                                             

    

    Explanation


  2. Message

    The norm log files show a lot senones that haev no observations during
   training.

    

    Explanation

    
This can only occur if all the utterances covering these
   senones die during the training. Alternately, you may have used
   decision trees that were not built using your training data. It is
   absolutely necessary to build trees using your training data, or if
   you have to take short cuts, to make sure that all the senones that
   you are training for are well represented in your training data. Otherwise
   you will have senones with zero counts and the decoder begins to
   do strange things. Even if you remove these senones from the mdef
   file for decoding, it does not ensure that the rest of the models
   have been trained properly.


    







Error messages

    

  1. Message

    

    

    Explanation

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    







Checking the model parameters


The norm executable generates four files which constitute the complete set of
model parameters. These files contain the


All of these are binary files. After the norm has successfully run through, check each of these files
using the printp tool provided with the SPHINX package. This tool prints each
of these files in ascii format. In order to 
know what to look for, it is first necessary to understand the arrangement of
model parameters in each of these files. Semi-continuous model parameter
files have the following arrangements:




Assuming that you are working with the standard 4-stream feature set where
each stream is represented by 256 codewords, here is a list of what you must look for when checking these files:
















IS MY TRAINING OK?







I've just run the executable quick_count




Structure of the correct log file (the lines in red are explanations)



Total no. of words = 3423
Num_Phones = 47, Num_Words = 3423
Sil_Index = 0
#. of cxt-indep phones = 47
within-word triphone finished in 0.0166667 secs.
43 phones can begin a word
46 phones can end a word
83 single-phone words
*(*,*)s finished in 0.3 secs
b/e triphones finished in 0.183333 secs
used       2.90 sec




Warnings

    

  1. Message

    

    Explanation







Error messages

    

  1. Message

    

    Explanation

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    
















IS MY TRAINING OK?







I've just run the executable param_cnt




Structure of the correct log file (the lines in red are explanations)



$SphinxTrain/bin/param_cnt \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef \
 -ts2cbfn .semi. \
 -ctlfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.ctl \
 -lsnfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.newtranscripts \
 -dictfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign \
 -fdictfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign \
 -segdir dummy \
 -paramtype phone

[Switch]   [Default] [Value]
-moddeffn            /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
-ts2cbfn             .semi. 
-ctlfn               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.ctl
-part                       
-npart                      
-nskip                      
-runlen                     
-lsnfn               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.newtranscripts
-dictfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign
-fdictfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign
-segdir              dummy  
-segext    v8_seg    v8_seg 
-paramtype state     phone  
INFO: ../corpus.c(1270): Will process all remaining utts starting at 0
INFO: ../main.c(87): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 97344 total models defined (47 base, 97297 tri)
INFO: ../model_def_io.c(564): 584064 total states
INFO: ../model_def_io.c(565): 486720 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
INFO: ../main.c(98): Reading .semi.
INFO: ../main.c(132): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign
INFO: ../lexicon.c(207): 3422 entries added from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign
INFO: ../main.c(143): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign
INFO: ../lexicon.c(207): 3 entries added from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign
INFO: ../param_cnt.c(65): Scanning corpus
[1000] 




Warnings

    

  1. Message

    

    Explanation







Error messages

    

  1. Message

    

    Explanation

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    
















IS MY TRAINING OK?







I've just run the executable make_quests




Structure of the correct log file (the lines in red are explanations)




Warnings

    

  1. Message

    

    Explanation







Error messages

    

  1. Message

    -varfn
{path]/model_parameters/new_fe.ci_continuous/variances \
 -mixwfn
[path]/model_parameters/new_fe.ci_continuous/mixture_weights \
 -npermute 168 \
 -niter 0 \
 -qstperstt 20 \
.....
.....
.....
INFO: ../s3gau_io.c(128): Read
/sphx_train/hub97/training/model_parameters/new_fe.ci_continuous/means
[153x1x1 array]
INFO: ../s3gau_io.c(128): Read
/sphx_train/hub97/training/model_parameters/new_fe.ci_continuous/variances
[153x1x1 array]
FATAL_ERROR: "../ckd_alloc.c", line 109: ckd_calloc_2d failed for caller at
../main.c(186) at ../ckd_alloc.c(110)

    

    Explanation
make_quests searches 2^npermute combinations several times for the
optimal clustering of states. For this, it has to store 2^npermute
values (for the comparison). So, setting -npermute to anything greater
than 8 or 10 makes the program very slow, and anything over 28 will
make the program fail. We usually use a value of 8.

    

  2. Message

    segmentation fault (or very bad linguistic questions after make_quests
has run through)

    

    Explanation

    
Use the printp tool and check to see if any of your CI model parameters
are zero. If they are, then probably the phone(s) for which you tried to train
an HMM was not seen in the training data. You can't train what you don't
have examples for, so remove the phone(s) from your phonelist and retrain the
models.


    

    

  3. Message

    

    Explanation

    




Unexpected stops

    

  1. Stop at

    

    Explanation

    

















IS MY TRAINING OK?







I've just run the executable bldtree




Structure of the correct log file (the lines in red are explanations)



$SphinxTrain/bin/bldtree \
 -treefn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned/JH-0.dtree \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.untied.mdef \
 -mixwfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights \
 -ts2cbfn .semi. \
 -mwfloor 1e-30 \
 -psetfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions \
 -phone JH \
 -state 0 \
 -stwt 1.0 0.3 0.1 0.01 0.001 \
 -ssplitmin 1 \
 -ssplitmax 5 \
 -ssplitthr 0 \
 -csplitmin 1 \
 -csplitmax 500 \
 -csplitthr 0

[Switch]   [Default] [Value]
-treefn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned/JH-0.dtree
-moddeffn            /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.untied.mdef
-ts2cbfn   .semi.    .semi. 
-meanfn                     
-varfn                      
-varfloor  0.00001   1.000000e-05
-cntthresh 0.00001   1.000000e-05
-mixwfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights
-psetfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
-phone               JH     
-state               0      
-mwfloor   1e-4      1.000000e-30
-stwt                1.0 0.3 0.1 0.01 0.001
-ssplitthr 8e-4      0.000000e+00
-ssplitmin 1         1      
-ssplitmax 5         5      
-csplitthr 8e-4      0.000000e+00
-csplitmin 1         1      
-csplitmax 100       500    
INFO: ../main.c(150): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.untied.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 6859 total models defined (47 base, 6812 tri)
INFO: ../model_def_io.c(564): 41154 total states
INFO: ../model_def_io.c(565): 34295 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
INFO: ../main.c(188): Building trees for [JH AE AX i] through [JH UW IY i]
INFO: ../main.c(216): Covering states |[16500 16694]| == 195
INFO: ../main.c(223): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights
INFO: ../s3mixw_io.c(151): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights [195x4x256 array]
INFO: ../main.c(261): 39 of 39 models have observation count greater than 0.000010
INFO: ../main.c(65): nrm stwt: 0.709 0.213 0.071 0.007 0.001 
INFO: ../main.c(370): 39-class entropy: 9.277157e+03
INFO: ../main.c(493): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
INFO: ../main.c(530): 256 total simple questions (248 phone; 8 word bndry)

This means that you have 248/4 phone groupings
in your linguistic questions file. Every grouping serves as 4 questions, since
it can be applied to the left context, the right context, and negated and
applied to either contexts. You also have 8/2 word boundary questions in
your linguistic questions file, each of which can be applied to either context,
effectively giving you 8 word boundary questions.
INFO: ../main.c(532): 82 Left Only questions, and 82 Right Only questions

Of the 248/4 phone groupings, since you
already have context-specific questions in your linguistic questions file
(in this case), you have 82 questions that can be applied to the left context,
and 82 that can be applied to the right context. They can also of course be
negated and applied.
INFO: ../dtree.c(1492): Final simple tree

The root node is first split down to
ssplitmax number of leaves using the questions (and their simple negations)
The leaves are then exhaustively partitioned into two groups. The
question corresponding to the best partition is composed of questions
corresponding to these two leaves. It is called a complex question. Thus,
a simple tree is first built, and that yields a complex question corresponding
to the root node of that tree. The procedure is then applied recursively to
the leaves resulting from the branching of the node till there are no
more leaves to split (each leaf has only one context). The complex questions
are written into the tree file.

|( QUESTION0 -1 1.514e+04 1.025e+03 7.812e+02

The "|" indicates that a simple tree
is being built. The first string after ( is the name of the simple question
that resulted inthe best split of the root node. Following that is
the context, with -1 standing for the left context, 1 for the right context
and 0 for a wordbounday. Here the context is -1. Following that is the
likelihood associated with the root node. The next number is the *increase*
in likelihood after the root node was split (using that question), and the
final number is the count associated with the node (see the technical section
of this manual for further explanantions)
|    ( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02
|        ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02
|            ( - 2.574e+03 9 0 QUESTION4_1_L -1 2.751e+02 1.714e+02)
|            ( - 2.214e+03 9 0 QUESTION1 -1 2.001e+02 1.444e+02))
|        ( - 1.613e+03 4 0 QUESTION3_8_L -1 1.565e+02 1.033e+02))
|    ( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02
|        ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02
|            ( - 2.526e+03 1 0)
|            ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01))
|        ( - 2.691e+03 9 0 QUESTION0_8_R 1 2.637e+02 1.447e+02)))

The simple tree is now made. The following
lines beginning with s> are simply a repetition of the above information
s>    ( QUESTION0 -1 1.514e+04 1.025e+03 7.812e+02
s>        ( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02
s>            ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02
s>                ( - 2.574e+03 9 0 QUESTION4_1_L -1 2.751e+02 1.714e+02)
s>                ( - 2.214e+03 9 0 QUESTION1 -1 2.001e+02 1.444e+02))
s>            ( - 1.613e+03 4 0 QUESTION3_8_L -1 1.565e+02 1.033e+02))
s>        ( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02
s>            ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02
s>                ( - 2.526e+03 1 0)
s>                ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01))
s>            ( - 2.691e+03 9 0 QUESTION0_8_R 1 2.637e+02 1.447e+02)))INFO: ../dtree.c(1319): Comp split 0

Compound split 0 is completed, the root node
will now be split and in the following, one simple tree will be built for
each resulting leaf. The string ": Final simple tree" is written before
each simple tree is built. After the two simple trees are built,
the one that resulted in the higer likelihood increase will be chosen
for a further split, and two simple trees will be built for that. The
one that is split will contribute to the second compound question and
will be the second compund split. 
INFO: ../dtree.c(1492): Final simple tree
|( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02
|    ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02
|        ( - 2.526e+03 1 0)
|        ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01))
|    ( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02
|        ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01
|            ( - 1.084e+03 2 0 SILENCE -1 6.989e+01 6.443e+01)
|            ( - 2.512e+02 3 0 QUESTION0_9_R 1 6.235e+01 2.025e+01))
|        ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01
|            ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01)
|            ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01))))
s>    ( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02
s>        ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02
s>            ( - 2.526e+03 1 0)
s>            ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01))
s>        ( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02
s>            ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01
s>                ( - 1.084e+03 2 0 SILENCE -1 6.989e+01 6.443e+01)
s>                ( - 2.512e+02 3 0 QUESTION0_9_R 1 6.235e+01 2.025e+01))
s>            ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01
s>                ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01)
s>                ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01))))INFO: ../dtree.c(1492): Final simple tree
|( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02
|    ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02
|        ( QUESTION4_1_L -1 2.574e+03 2.751e+02 1.714e+02
|            ( - 8.883e+02 6 0 QUESTION2 -1 1.307e+02 6.173e+01)
|            ( - 1.410e+03 3 0 QUESTION0_8_R 1 1.033e+02 1.097e+02))
|        ( QUESTION1 -1 2.214e+03 2.001e+02 1.444e+02
|            ( - 7.658e+02 4 0 QUESTION1_8_R 1 9.251e+01 5.624e+01)
|            ( - 1.248e+03 5 0 QUESTION0_8_R 1 1.364e+02 8.813e+01)))
|    ( QUESTION3_8_L -1 1.613e+03 1.565e+02 1.033e+02
|        ( - 5.607e+02 2 0 QUESTION3_9_L -1 6.001e+01 4.016e+01)
|        ( - 8.963e+02 2 0 SILENCE 1 7.711e+01 6.314e+01)))
s>    ( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02
s>        ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02
s>            ( QUESTION4_1_L -1 2.574e+03 2.751e+02 1.714e+02
s>                ( - 8.883e+02 6 0 QUESTION2 -1 1.307e+02 6.173e+01)
s>                ( - 1.410e+03 3 0 QUESTION0_8_R 1 1.033e+02 1.097e+02))
s>            ( QUESTION1 -1 2.214e+03 2.001e+02 1.444e+02
s>                ( - 7.658e+02 4 0 QUESTION1_8_R 1 9.251e+01 5.624e+01)
s>                ( - 1.248e+03 5 0 QUESTION0_8_R 1 1.364e+02 8.813e+01)))
s>        ( QUESTION3_8_L -1 1.613e+03 1.565e+02 1.033e+02
s>            ( - 5.607e+02 2 0 QUESTION3_9_L -1 6.001e+01 4.016e+01)
s>            ( - 8.963e+02 2 0 SILENCE 1 7.711e+01 6.314e+01)))INFO: ../dtree.c(1319): Comp split 1

The second compound split is completed
INFO: ../dtree.c(1492): Final simple tree
|( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02
|    ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01
|        ( SILENCE -1 1.084e+03 6.989e+01 6.443e+01
|            ( - 9.718e+02 1 0)
|            ( - 4.220e+01 1 0))
|        ( QUESTION0_9_R 1 2.512e+02 6.235e+01 2.025e+01
|            ( - 8.426e+01 2 0)
|            ( - 1.046e+02 1 0)))
|    ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01
|        ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01)
|        ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01)))
s>    ( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02
s>        ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01
s>            ( SILENCE -1 1.084e+03 6.989e+01 6.443e+01
s>                ( - 9.718e+02 1 0)
s>                ( - 4.220e+01 1 0))
s>            ( QUESTION0_9_R 1 2.512e+02 6.235e+01 2.025e+01
s>                ( - 8.426e+01 2 0)
s>                ( - 1.046e+02 1 0)))
s>        ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01
s>            ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01)
s>            ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01)))INFO: ../dtree.c(1492): Final simple tree
|( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02
|    ( - 2.526e+03 1 0)
|    ( QUESTION0_11_R 1 5.633e+02 9.871e+01 3.687e+01
|        ( QUESTION4_0_L -1 2.358e+02 5.519e+01 1.902e+01
|            ( - 4.532e+01 2 0 WDBNDRY_B 0 1.694e+01 6.000e+00)
|            ( - 1.353e+02 1 0))
|        ( QUESTION1 1 2.288e+02 4.944e+01 1.786e+01
|            ( - 9.277e+01 2 0 WDBNDRY_B 0 2.191e+01 8.992e+00)
|            ( WDBNDRY_E 0 8.660e+01 2.737e+01 8.863e+00
|                ( - 3.073e+01 1 0)
|                ( - 2.850e+01 1 0)))))
s>    ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02
s>        ( - 2.526e+03 1 0)
s>        ( QUESTION0_11_R 1 5.633e+02 9.871e+01 3.687e+01
s>            ( QUESTION4_0_L -1 2.358e+02 5.519e+01 1.902e+01
s>                ( - 4.532e+01 2 0 WDBNDRY_B 0 1.694e+01 6.000e+00)
s>                ( - 1.353e+02 1 0))
s>            ( QUESTION1 1 2.288e+02 4.944e+01 1.786e+01
s>                ( - 9.277e+01 2 0 WDBNDRY_B 0 2.191e+01 8.992e+00)
s>                ( WDBNDRY_E 0 8.660e+01 2.737e+01 8.863e+00
s>                    ( - 3.073e+01 1 0)
s>                    ( - 2.850e+01 1 0)))))
INFO: ../dtree.c(1319): Comp split 2

The third compound split is completed.
For an explanation of this algorithm, click here

INFO: ../dtree.c(1492): Final simple tree

************* and so on till cssplitmax compound splits are done*******
*********usually cssplitmax is set to a very high number, so the******
**** compound splits stop when there is nothing more to split*********

INFO: ../dtree.c(1492): Final simple tree
|( WDBNDRY_B 0 9.277e+01 2.191e+01 8.992e+00
|    ( - 5.843e+01 1 0)
|    ( - 1.244e+01 1 0))
s>    ( WDBNDRY_B 0 9.277e+01 2.191e+01 8.992e+00
s>        ( - 5.843e+01 1 0)
s>        ( - 1.244e+01 1 0))INFO: ../dtree.c(1319): Comp split 27
INFO: ../dtree.c(1319): Comp split 28
INFO: ../dtree.c(1451): stop. leaf nodes are specific
INFO: ../dtree.c(1492): Final simple tree
|( WDBNDRY_B 0 1.452e+02 3.611e+01 1.582e+01
|    ( - 5.028e+01 1 0)
|    ( - 5.877e+01 1 0))
s>    ( WDBNDRY_B 0 1.452e+02 3.611e+01 1.582e+01
s>        ( - 5.028e+01 1 0)
s>        ( - 5.877e+01 1 0))INFO: ../dtree.c(1319): Comp split 29
INFO: ../dtree.c(1319): Comp split 30
INFO: ../dtree.c(1319): Comp split 31
INFO: ../dtree.c(1319): Comp split 32
INFO: ../dtree.c(1319): Comp split 33
INFO: ../dtree.c(1319): Comp split 34
INFO: ../dtree.c(1319): Comp split 35
INFO: ../dtree.c(1319): Comp split 36
INFO: ../dtree.c(1319): Comp split 37
INFO: ../dtree.c(1322): stop. leaf nodes are specific




Warnings

    

  1. Message

    

    Explanation







Error messages

    

  1. Message

    
In the file (experimentname).buidtree.(phone).(state).log:

    FATAL_ERROR: "../ckd_alloc.c", line 55: Calloc failed from ../two_class.c(95)

    

    Explanation

    
The cd-untied models are probably untrained or very badly trained. Check
the cd-untied training logfiles.

    

  2. Message

    

    'WARNING: "../main.c", line 183: No triphones involving +SMACK+
FATAL_ERROR: "../ckd_alloc.c", line 55: Calloc failed from ../main.c(668)'.

    

    Explanation

    
No triphones were trained for the phone +SMACK+. In this case, the
phone (+SMACK+) is a filler phone, for which we do not train triphones
anyway. We do not build trees for filler phones. So if bldtree generates
this error message and does notoutput any tree for a filler phone,
it's okay - just ignore this.

    

  3. Message

    

    FATAL_ERROR: "../main.c", line 276: Fewer state weights than states'.

    

    Explanation

    
If you are building N-state HMMs, then you must supply one weight per state
to the flag -stwt for bldtree. For example, the setting for 5-state HMMs
could be

    -stwt     1.0 0.3 0.1 0.01 0.001

    
Make sure that the number of weights are equal to the number of states.

    

  4. Message

    

    Explanation

    




Unexpected stops

    

  1. Stop at

    INFO: main.c(103): nrm stwt: 0.709 0.213 0.071 0.007 0.001
INFO: main.c(408): 104-class entropy: 4.708714e+04
INFO: main.c(531): Reading: /path/model_architecture/tongues.tree_questions
INFO: main.c(568): 196 total simple questions (188 phone; 8 word bndry)
INFO: main.c(570): 0 Left Only questions, and 0 Right Only questions
INFO: dtree.c(1353): Comp split 0
INFO: dtree.c(1362): stop. b_n->wt_ent_dec (0.000e+00) <= 0   

    

    Explanation

    



  2. The structure of a tree file

    

    n_node 75
0 1 2 1.024970e+03 7.812383e+02 ((!QUESTION0 -1))
1 3 4 7.827493e+02 3.621280e+02 ((!QUESTION12 -1))
3 13 14 2.715968e+02 1.446638e+02 ((QUESTION0_8_R 1 !QUESTION0_11_R 1 !QUESTION0_9_R 1)(!QUESTION0_8_R 1))
13 19 20 1.588734e+02 7.022660e+01 ((QUESTION0_19_R 1 SILENCE -1)(!QUESTION0_19_R 1 !QUESTION0_8_R 1 !SILENCE -1))
19 43 44 6.583110e+01 4.498003e+01 ((!SILENCE -1))
43 - - 3.678522e+01 6.000000e+00
44 - - 5.835646e+02 3.898003e+01
20 41 42 7.375260e+01 2.524657e+01 ((!QUESTION0_18_R 1))
41 67 68 3.048508e+01 1.324949e+01 ((!SILENCE -1))
67 - - 1.570110e+01 3.000000e+00
68 - - 1.045684e+02 1.024949e+01
42 - - 1.031438e+02 1.199708e+01
14 31 32 9.854553e+01 7.443717e+01 ((!QUESTION0_9_R 1 SILENCE -1))
31 - - 9.718422e+02 5.943530e+01
32 55 56 4.939473e+01 1.500187e+01 ((!SILENCE -1))
55 - - 4.219557e+01 4.999490e+00
56 - - 8.426309e+01 1.000238e+01
4 9 10 2.908346e+02 2.174642e+02 ((!QUESTION3_0_L -1))
9 29 30 9.870528e+01 3.687373e+01 ((!QUESTION0_11_R 1))
29 53 54 4.943643e+01 1.785568e+01 ((!QUESTION1 1))
53 69 70 2.736811e+01 8.863489e+00 ((!WDBNDRY_E 0))
69 - - 2.849812e+01 4.000000e+00
70 - - 3.072878e+01 4.863489e+00
54 71 72 2.190642e+01 8.992188e+00 ((!WDBNDRY_B 0))
71 - - 1.244149e+01 2.000000e+00
72 - - 5.842520e+01 6.992188e+00
30 49 50 5.518961e+01 1.901805e+01 ((QUESTION4_0_L -1))
49 73 74 1.694358e+01 5.999987e+00 ((!WDBNDRY_B 0))
73 - - 1.270456e+01 3.000002e+00
74 - - 1.567007e+01 2.999985e+00
50 - - 1.352641e+02 1.301806e+01
10 - - 2.525817e+03 1.805905e+02
2 5 6 4.985232e+02 4.191103e+02 ((QUESTION0 1 QUESTION3_11_L -1))
5 11 12 2.751202e+02 1.714405e+02 ((QUESTION4_1_L -1))
11 21 22 1.381941e+02 6.173034e+01 ((QUESTION2 -1 !QUESTION0_8_R 1)(!QUESTION1 -1))
21 57 58 4.674642e+01 2.281793e+01 ((QUESTION0_8_R 1))
57 63 64 3.611139e+01 1.581815e+01 ((!WDBNDRY_B 0))
63 - - 5.877301e+01 8.818151e+00
64 - - 5.028220e+01 6.999999e+00
58 - - 3.269149e+01 6.999775e+00
22 37 38 8.078258e+01 3.891241e+01 ((!QUESTION2 -1))
37 65 66 3.547732e+01 1.299995e+01 ((!WDBNDRY_B 0))
65 - - 4.989049e+01 6.000001e+00
66 - - 5.591455e+01 6.999952e+00
38 - - 3.034155e+02 2.591245e+01
12 27 28 1.032965e+02 1.097102e+02 ((!QUESTION0_8_R 1))
27 51 52 5.351306e+01 2.600765e+01 ((!WDBNDRY_E 0))
51 - - 1.894188e+02 1.800761e+01
52 - - 6.225969e+01 8.000047e+00
28 - - 1.001935e+03 8.370250e+01
6 7 8 3.608895e+02 2.476698e+02 ((!WDBNDRY_E 0))
7 15 16 2.078070e+02 1.443672e+02 ((!QUESTION1 -1 QUESTION0_8_R 1)(!QUESTION1 -1 !QUESTION0_8_R 1 !QUESTION6 1))
15 23 24 1.370261e+02 7.212400e+01 ((QUESTION0_8_R 1 !QUESTION5 -1 WDBNDRY_B 0)(!QUESTION0_8_R 1))
23 45 46 6.179547e+01 2.710147e+01 ((!QUESTION0_8_R 1))
45 - - 2.103741e+02 1.905201e+01
46 - - 5.734571e+01 8.049463e+00
24 39 40 8.045180e+01 4.502253e+01 ((!WDBNDRY_B 0))
39 - - 1.483478e+02 1.800076e+01
40 - - 3.054461e+02 2.702177e+01
16 25 26 1.264272e+02 7.224318e+01 ((QUESTION0_11_R 1 !QUESTION1_8_R 1 !WDBNDRY_B 0)(!QUESTION0_11_R 1))
25 33 34 8.898907e+01 4.871862e+01 ((!QUESTION0_11_R 1))
33 59 60 4.497908e+01 2.671408e+01 ((!WDBNDRY_B 0))
59 - - 1.028076e+02 1.071100e+01
60 - - 1.498041e+02 1.600309e+01
34 - - 2.196730e+02 2.200454e+01
26 47 48 5.541730e+01 2.352456e+01 ((!WDBNDRY_B 0))
47 - - 1.090463e+02 1.200061e+01
48 - - 1.080980e+02 1.152395e+01
8 17 18 1.678853e+02 1.033026e+02 ((!QUESTION3_8_L -1 SILENCE 1))
17 - - 7.428963e+02 5.414283e+01
18 35 36 8.536245e+01 4.915981e+01 ((!QUESTION3_10_L -1))
35 61 62 4.037777e+01 1.399997e+01 ((!SILENCE 1))
61 - - 7.629050e+01 8.999999e+00
62 - - 3.061310e+01 4.999975e+00
36 - - 4.700490e+02 3.515984e+01


    















IS MY TRAINING OK?







I've just run the executable prunetree




Structure of the correct log file (the lines in red are explanations)



$SphinxTrain/bin/prunetree \
 -itreedir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned \
 -nseno 4000 \
 -otreedir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000 \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef \
 -psetfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions \
 -minocc 0

[Switch]  [Default] [Value]
-moddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
-psetfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
-itreedir           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned
-otreedir           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000
-nseno              4000   
-minocc   0.0       0.000000e+00
INFO: ../main.c(52): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 97344 total models defined (47 base, 97297 tri)
INFO: ../model_def_io.c(564): 584064 total states
INFO: ../model_def_io.c(565): 486720 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
INFO: ../main.c(58): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
INFO: ../main.c(188): AA-0	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-1	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-2	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-3	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-4	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AE-0	102 [0 < 0.000000e+00]
INFO: ../main.c(188): AE-1	102 [0 < 0.000000e+00]

Starting from the right of the first line 
above, the number 0.000000e+00 is the value you have given to the flag
-minocc in the prunetree executable. This flag allows you to specify the
minimum number of counts that must be associated with each leaf of a tree.
The count associated with any node or leaf is the estimated number of observations
 in that node or leaf. In first stage of pruning, the trees are pruned 
entirely with regard to these counts. The number 0.000000e+00 indicates
that the count was set to zero in this particular case. The experession
[0 < 0.000000e+00] means that there were zero leaves which had counts less
than zero. The number 65 means that the tree, after zero nodes were
pruned out, was left with 65 leaves. The string AA-0 indicates that
this information  is about the decision tree for the 1st state of the
phone AA
INFO: ../main.c(188): AE-2	102 [0 < 0.000000e+00]
INFO: ../main.c(188): AE-3	102 [0 < 0.000000e+00]
********** and so on till the last tree for the last phone ******
INFO: ../main.c(188): ZH-3	2 [0 < 0.000000e+00]
INFO: ../main.c(188): ZH-4	2 [0 < 0.000000e+00]
INFO: ../main.c(205): Prior to pruning n_seno= 25725
INFO: ../main.c(213): n_twig= 9529
INFO: ../main.c(242): Pruning 21725 nodes

The next stage of pruning begins here. In this
stage, trees which have been first pruned with regard to the minimum
occurance threshold given, are pruned with regard to entropy
INFO: ../main.c(274): Root node extracted (ZH 4) from heap
INFO: ../main.c(274): Root node extracted (ZH 3) from heap

The first line means that all leaves of the tree
for the 5th state of the phone ZH got pruned out and you now have a tree
with a single or "root" node. This implies that there will be a single
tied state (or senone) representing the 5th state of all triphones for the 
phone ZH. You can confirm this by looking at the state-ids in the tied state
model definition file which you will make using these pruned trees for CD-tied
training   
INFO: ../main.c(274): Root node extracted (ZH 0) from heap
INFO: ../main.c(274): Root node extracted (ZH 2) from heap
INFO: ../main.c(274): Root node extracted (ZH 1) from heap
INFO: ../main.c(274): Root node extracted (OY 3) from heap
INFO: ../main.c(274): Root node extracted (OY 2) from heap
INFO: ../main.c(274): Root node extracted (OY 4) from heap
INFO: ../main.c(274): Root node extracted (OY 1) from heap
INFO: ../main.c(312): AA-0	33
INFO: ../main.c(312): AA-1	31

The first line means that after the 
entropy-based pruning, the tree for the 1st state of the phone AA was left 
with 33 leaves. The first state of all triphones for the phone AA, therefore, 
will be represented by 33 different senone ids
************* and so on till the last state of the last phone *********
INFO: ../main.c(312): ZH-0	1
INFO: ../main.c(312): ZH-1	1
INFO: ../main.c(312): ZH-2	1
INFO: ../main.c(312): ZH-3	1
INFO: ../main.c(312): ZH-4	1




Warnings

    

  1. Message

    

    Explanation







Error messages

    

  1. Message

    

    Explanation

    




Unexpected stops and obscure indications of a problem

    

  1. Stop at
Here are some entries from a  logfile that has been written during a seemingly
smooth run of prunetree. There are no warnings or error messages, but something is very wrong..

    
-otreedir           /path/trees/tongues.6000
-nseno              6000   
-minocc   0.0       0.000000e+00
INFO: main.c(87): Reading: /path/model_architecture/tongues.alltriphones.mdef
INFO: model_def_io.c(598): Model definition info:
INFO: model_def_io.c(599): 74796 total models defined (47 base, 74749 tri)
INFO: model_def_io.c(600): 448776 total states
INFO: model_def_io.c(601): 373980 total tied states
INFO: model_def_io.c(602): 235 total tied CI states
INFO: model_def_io.c(603): 47 total tied transition matrices
INFO: model_def_io.c(604): 6 max state/model
INFO: model_def_io.c(605): 20 min state/model
INFO: main.c(93): Reading: /path/model_architecture/tongues.tree_questions
INFO: main.c(223): AA-0	1 [0 < 0.000000e+00]
INFO: main.c(223): AA-1	1 [0 < 0.000000e+00]
INFO: main.c(223): AA-2	1 [0 < 0.000000e+00]
INFO: main.c(223): AA-3	1 [0 < 0.000000e+00]
INFO: main.c(223): AA-4	1 [0 < 0.000000e+00]
INFO: main.c(223): AE-0	1 [0 < 0.000000e+00]
INFO: main.c(223): AE-1	1 [0 < 0.000000e+00]
INFO: main.c(223): AE-2	1 [0 < 0.000000e+00]
INFO: main.c(223): AE-3	1 [0 < 0.000000e+00]
INFO: main.c(223): AE-4	1 [0 < 0.000000e+00]
******** and so on till *******************************

INFO: main.c(223): Z-3	1 [0 < 0.000000e+00]
INFO: main.c(223): Z-4	1 [0 < 0.000000e+00]
INFO: main.c(223): ZH-0	1 [0 < 0.000000e+00]
INFO: main.c(223): ZH-1	1 [0 < 0.000000e+00]
INFO: main.c(223): ZH-2	1 [0 < 0.000000e+00]
INFO: main.c(223): ZH-3	1 [0 < 0.000000e+00]
INFO: main.c(223): ZH-4	1 [0 < 0.000000e+00]
INFO: main.c(240): Prior to pruning n_seno= 200
WARNING: "main.c", line 244: n_seno_wanted= 6000, but only 200 defined by trees
INFO: main.c(248): n_twig= 0
INFO: main.c(347): AA-0	1
INFO: main.c(347): AA-1	1
INFO: main.c(347): AA-2	1
INFO: main.c(347): AA-3	1
INFO: main.c(347): AA-4	1
INFO: main.c(347): AE-0	1
INFO: main.c(347): AE-1	1
INFO: main.c(347): AE-2	1
INFO: main.c(347): AE-3	1
******** and so on till *******************************

INFO: main.c(347): Z-2	1
INFO: main.c(347): Z-3	1
INFO: main.c(347): Z-4	1
INFO: main.c(347): ZH-0	1
INFO: main.c(347): ZH-1	1
INFO: main.c(347): ZH-2	1
INFO: main.c(347): ZH-3	1
INFO: main.c(347): ZH-4	1


    

    Explanation

    
In the logfile above, you see that in the first stage of pruning, every single
tree had only one leaf (the root node). Obviously then, after the second stage
of pruning, there was again only one leaf in every tree. The first set of
ones indicates that none of the trees ever branched - and that could be
due to many reasons. A tree will not branch if the linguistic questions simply
repeat with different names in your linguistic questions file, or are
bad in some other bizarre manner. First check your linguistic questions file...
If that checks out ok, then look at the mixture weights. Are they all zero?
Are they all equal? Does the last norm logfile report any errors or warnings?
If it does, see if you can trace back the problem from there. If the 
linguistic questions are back, the trace can begin from the logfile for
make_quests.






  • The structure of a pruned tree file

    

    n_node 9
0 1 4 1.024970e+03 7.812383e+02 ((!QUESTION0 -1))
1 2 3 7.827493e+02 3.621280e+02 ((!QUESTION12 -1))
2 - - 0.000000e+00 1.446638e+02
3 - - 0.000000e+00 2.174642e+02
4 5 6 4.985232e+02 4.191103e+02 ((QUESTION0 1 QUESTION3_11_L -1))
5 - - 0.000000e+00 1.714405e+02
6 7 8 3.608895e+02 2.476698e+02 ((!WDBNDRY_E 0))
7 - - 0.000000e+00 1.443672e+02
8 - - 0.000000e+00 1.033026e+02

    


    

    








    

    IS MY TRAINING OK?
    

    


    

    I've just run the executable tiestate

    

    
Structure of the correct log file (the lines in red are explanations)

    

    $SphinxTrain/bin/tiestate \
 -imoddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef \
 -omoddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.4000.mdef \
 -treedir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000 \
 -psetfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions

[Switch]   [Default] [Value]
-imoddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
-omoddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.4000.mdef
-treedir             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000
-psetfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 78204 total models defined (47 base, 78157 tri)
INFO: ../model_def_io.c(564): 469224 total states
INFO: ../model_def_io.c(565): 391020 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
INFO: ../main.c(68): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
INFO: ../main.c(86): AA-0: offset 235
INFO: ../main.c(86): AA-1: offset 252

    
The first line says that with respect to the first state of the first CI phone,
the first tied state of the first state of the phone AA (which happens to be
the first phone here) is 235. For the second state of AA, the offset is 252.
This means that ALL the triphones of AA have their first states assigned
numerical ids which lie between 235 and 252.
INFO: ../main.c(86): AA-2: offset 268
INFO: ../main.c(86): AA-3: offset 283
INFO: ../main.c(86): AA-4: offset 300
INFO: ../main.c(86): AE-0: offset 317
*********** and so on till the last state of the last phone ********

INFO: ../main.c(86): ZH-3: offset 4233
INFO: ../main.c(86): ZH-4: offset 4234
INFO: ../main.c(108): n_seno= 4235


    

    
Warnings

      

    1. Message

      

      Explanation

    

    


    
Error messages

      

    1. Message

      

      Explanation

      

    

    
Unexpected stops and obscure indications of a problem

      

    1. Stop at

      /path/bin/tiestate \
 -imoddeffn /path/model_architecture/tongues.alltriphones.mdef \
 -omoddeffn /path/model_architecture/tongues.6000.mdef \
 -treedir /path/trees/tongues.6000 \
 -psetfn /path/model_architecture/tongues.tree_questions

[Switch]   [Default] [Value]
-imoddeffn           /path/model_architecture/tongues.alltriphones.mdef
-omoddeffn           /path/model_architecture/tongues.6000.mdef
-treedir             /path/trees/tongues.6000
-psetfn              /path/model_architecture/tongues.tree_questions
INFO: model_def_io.c(598): Model definition info:
INFO: model_def_io.c(599): 74796 total models defined (47 base, 74749 tri)
INFO: model_def_io.c(600): 448776 total states
INFO: model_def_io.c(601): 373980 total tied states
INFO: model_def_io.c(602): 235 total tied CI states
INFO: model_def_io.c(603): 47 total tied transition matrices
INFO: model_def_io.c(604): 6 max state/model
INFO: model_def_io.c(605): 20 min state/model
INFO: main.c(102): Reading: /path/model_architecture/tongues.tree_questions
INFO: main.c(120): AA-0: offset 235
INFO: main.c(120): AA-1: offset 236


      
The first line says that with respect to the first state of the first CI phone,
the first tied state of the first state of the phone AA (which happens to be 
the first phone here) is 235. For the second state of AA, the offset is 236. 
This means that ALL the triphones of AA have their first states mapped to 235. 
This can happen when the tree for the first state of AA has only one node. For
a phone like AA which is very common in the language, this is obviously
an error. The tree should have been deeper than one node. You have to
search the logfiles of bldtree to get to the source of this error)
INFO: main.c(120): AA-2: offset 237
INFO: main.c(120): AA-3: offset 238
INFO: main.c(120): AA-4: offset 239
INFO: main.c(120): AE-0: offset 240
****** and so on till the last state of the last phone **********

INFO: main.c(120): ZH-3: offset 433
INFO: main.c(120): ZH-4: offset 434
INFO: main.c(142): n_seno= 435
AA AA AA s 0 	-> 235

AA AA AA s 1 	-> 236

AA AA AA s 2 	-> 237

      
The mode of compilation turns on some compiler directives which cause
the program to print out extended log files. These lines may
have been printed either due to that or just due to some explicit
print in the program. In any case they are of zero use. 
These lines are simply indicating
the state-ids assigned to each triphone. The print has to be disabled in the
opensource version. )
**** and thousands of lines of garbage till ******
ZH UW AX i 4 	-> 434

      

      Explanation

      

    

    

    






    

    IS MY DECODING SETUP OK?
    

    
The first thing that you must ensure is that the feature files
are being correctly computed. 
If you are about to decode with existing models, then make sure that

    
Here's the next quick checklist: