INDEX

(This is under construction..)

  1. Tools for troubleshooting
  2. Is my training ok?
    1. Semi-continuous models
      I've just run the executable..........
    2. Continuous models
      I've just run the executable..........
  3. Is my decoding setup ok?

TOOLS FOR TROUBLESHOOTING

The two most useful executables used for troubleshooting are

  1. cepview:

    This can be use to view the binary feature file you have generated using the feature computation modules provided with SPHINX. You can display up to N dimensions of the feature vectors by using the flag value

    -d       N
    
    The output is in ascii as stdout, and can be piped into any file or program which accepts it.

    In the case of MFCCs, depending on the type of DCT implementation, the values that you will see for the first dimension of each vector might range from 0 to a few hundred. Values in the rest of the dimensions are usually smaller. If the numbers in the file are much bigger, or if you see the sequence "NaN" in the ascii output, it indicates that there is some problem with your audio data. The data could be byte swapped, or you may have computed features for the headers as well, or you may have had digital zeros in the recordings and may have forgotten to add a one-bit dither (or so) while computing the features etc.

  2. printp:

    This executable can be used for printing out the binary model paramter files in ascii format. Depending on which file you are want to look at, one or more of the following flags must be set appropriately:

    [Switch]     [Default] [Description] 
    -tmatfn                The transition matrix parameter file name
    -mixwfn                The mixture weight parameter file name
    -mixws                 Start id of mixing weight subinterval
    -mixwe                 End id of mixing weight subinterval
    -gaufn                 A Gaussian parameter file name (either for means or vars)
    -gaucntfn              A Gaussian parameter weighted vector file
    -regmatcntfn           MLLR regression matrix count file
    -moddeffn              The model definition file
    -lambdafn              The interpolation weight file
    -lambdamin   0         Print int. wt. >= this
    -lambdamax   1         Print int. wt. <= this
    -norm        yes       Print normalized parameters
    -sigfig      4         Number of significant digits in 'e' notation 
    


IS MY TRAINING OK?

If you are training models, even though the training seems to be going through smoothly, keep checking the logfiles being generated for possible problems, and also keep checking the various model architecture and model parameter files being generated. The sections that follow discuss how you can track down problems through these checks before you have wasted a lot of CPU and your own time on the training.

The first thing that you must ensure is that the feature files are being correctly computed. If you are about to train models, then make sure that

IS MY TRAINING OK?

I've just run the executable agg_seg

Structure of the correct log file (the lines in red are explanations)

$SphinxTrain/bin/agg_seg \
 -segdmpdirs /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 \
 -segdmpfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp \
 -segtype all \
 -ctlfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.fileids \
 -cepdir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//feat \
 -cepext feat \
 -ceplen 13 \
 -agc none \
 -cmn current \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -stride 1

[Switch]    [Default] [Value]
-segdmpdirs           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1
-segdmpfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp
-segidxfn                    
-segtype    st        all    
-cntfn                       
-ddcodeext  xcode     xcode  
-lsnfn                       
-sentdir                     
-sentext                     
-ctlfn                /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.fileids
-mllrctlfn                   
-mllrdir                     
-nskip      0         0      
-runlen     -1        -1     
-moddeffn                    
-ts2cbfn                     
-cb2mllrfn                   
-dictfn                      
-fdictfn                     
-segdir                      
-segext     v8_seg    v8_seg 
-ccodedir                    
-ccodeext   ccode     ccode  
-dcodedir                    
-dcodeext   d2code    d2code 
-pcodedir                    
-pcodeext   p3code    p3code 
-ddcodedir                   
-ddcodeext  xcode     xcode  
-cepdir               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//feat
-cepext     mfc       feat   
-ceplen     13        13     
-agc        max       none   
-cmn        current   current
-varnorm    no        no     
-silcomp    none      none   
-feat                 c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-cachesz    200       200    
-stride     1         1      
INFO: ../main.c(141): No lexical transcripts provided
INFO: ../corpus.c(1270): Will process all remaining utts starting at 0
INFO: ../main.c(245): Will produce FEAT dump
INFO: ../main.c(401): Writing frames to one file
INFO: ../agg_all_seg.c(124): [0]
INFO: ../agg_all_seg.c(124): [1000]
INFO: ../agg_all_seg.c(165): Wrote 1229629 frames to /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    areadfloat: path_to_cep/utt.mfc: can't alloc data
    ERROR: "../corpus.c", line 1460: MFCC read failed. Retrying after sleep...
    

    Explanation

    This happens when data are byte-swapped or there are very few frames in utterance. It also happens when your feature file is physically not present or is inaccessible/unreadable due to some reason.

Unexpected stops

  1. Stop at
    
    

    Explanation


IS MY TRAINING OK?

I've just run the executable kmeans

Structure of the correct log file (the lines in red are explanations)

$SphinxTrain/bin/kmeans_init \
 -gthobj single \
 -stride 1 \
 -ntrial 1 \
 -minratio 0.001 \
 -ndensity 256 \
 -meanfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/means \
 -varfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/variances \
 -reest no \
 -segdmpdirs /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 \
 -segdmpfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp \
 -ceplen 13 \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -agc none \
 -cmn current

[Switch]    [Default]          [Value]
-segdir                               
-segext     v8_seg             v8_seg 
-omoddeffn                            
-dmoddeffn                            
-ts2cbfn                              
-lsnfn                                
-dictfn                               
-fdictfn                              
-cbcntfn                              
-maxcbobs                             
-maxtotobs                            
-featsel                              
-ctlfn                                
-cepext     mfc                mfc    
-cepdir                               
-ceplen     13                 13     
-agc        max                none   
-cmn        current            current
-varnorm    no                 no     
-silcomp    none               none   
-feat       1s_12c_12d_3p_12dd c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-segdmpdirs                    /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1
-segdmpfn                      /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tongues3.dmp
-segidxfn                             
-fpcachesz  3000               3000   
-obscachesz 92                 92     
-ndensity                      256    
-ntrial     5                  1      
-minratio   0.01               1.000000e-03
-maxiter    100                100    
-mixwfn                               
-meanfn                        /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/means
-varfn                         /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/variances
-method     rkm                rkm    
-reest      yes                no     
-niter      20                 20     
-gthobj     state              single 
-stride     32                 1      
-runlen                               
-tsoff      0                  0      
-tscnt                                
-tsrngfn                              
-vartiethr  0                  0      
INFO: ../main.c(458): No mdef files.  Assuming 1-class init
INFO: ../main.c(1230): 1-class dump file
INFO: ../main.c(1268): Corpus 0: sz==1229629 frames
INFO: ../main.c(1277): Convergence ratios are abs(cur - prior) / abs(prior)
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 2.085017e-01 ...
INFO: ../kmeans.c(130): km iter [2] 3.839041e-02 ...
INFO: ../kmeans.c(130): km iter [3] 1.496473e-02 ...
INFO: ../kmeans.c(130): km iter [4] 8.121040e-03 ...
INFO: ../kmeans.c(130): km iter [5] 5.256977e-03 ...
INFO: ../kmeans.c(130): km iter [6] 3.857913e-03 ...
INFO: ../kmeans.c(130): km iter [7] 2.907482e-03 ...
INFO: ../kmeans.c(130): km iter [8] 2.227546e-03 ...
INFO: ../kmeans.c(130): km iter [9] 1.753199e-03 ...
INFO: ../kmeans.c(130): km iter [10] 1.408711e-03 ...
INFO: ../kmeans.c(130): km iter [11] 1.177712e-03 ...
INFO: ../kmeans.c(130): km iter [12] 1.001406e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 13 sqerr 3.289491e+05 conv_ratio 8.766461e-04
INFO: ../main.c(561): 	best-so-far sqerr = 3.289491e+05
INFO: ../main.c(211): alloc'ing 112Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 2.146148e-01 ...
INFO: ../kmeans.c(130): km iter [2] 3.472337e-02 ...
INFO: ../kmeans.c(130): km iter [3] 1.313343e-02 ...
INFO: ../kmeans.c(130): km iter [4] 7.271150e-03 ...
INFO: ../kmeans.c(130): km iter [5] 4.842221e-03 ...
INFO: ../kmeans.c(130): km iter [6] 3.540583e-03 ...
INFO: ../kmeans.c(130): km iter [7] 2.704598e-03 ...
INFO: ../kmeans.c(130): km iter [8] 2.159868e-03 ...
INFO: ../kmeans.c(130): km iter [9] 1.765137e-03 ...
INFO: ../kmeans.c(130): km iter [10] 1.490874e-03 ...
INFO: ../kmeans.c(130): km iter [11] 1.267887e-03 ...
INFO: ../kmeans.c(130): km iter [12] 1.080588e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 13 sqerr 1.172793e+06 conv_ratio 9.439682e-04
INFO: ../main.c(561): 	best-so-far sqerr = 1.172793e+06
INFO: ../main.c(211): alloc'ing 14Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 1.958686e-01 ...
INFO: ../kmeans.c(130): km iter [2] 7.782481e-02 ...
INFO: ../kmeans.c(130): km iter [3] 4.728356e-02 ...
INFO: ../kmeans.c(130): km iter [4] 3.346227e-02 ...
INFO: ../kmeans.c(130): km iter [5] 2.553782e-02 ...
INFO: ../kmeans.c(130): km iter [6] 1.942101e-02 ...
INFO: ../kmeans.c(130): km iter [7] 1.523729e-02 ...
INFO: ../kmeans.c(130): km iter [8] 1.252560e-02 ...
INFO: ../kmeans.c(130): km iter [9] 1.063390e-02 ...
INFO: ../kmeans.c(130): km iter [10] 9.017836e-03 ...
INFO: ../kmeans.c(130): km iter [11] 7.604756e-03 ...
INFO: ../kmeans.c(130): km iter [12] 6.461693e-03 ...
INFO: ../kmeans.c(130): km iter [13] 5.621540e-03 ...
INFO: ../kmeans.c(130): km iter [14] 4.818882e-03 ...
INFO: ../kmeans.c(130): km iter [15] 4.225533e-03 ...
INFO: ../kmeans.c(130): km iter [16] 3.757179e-03 ...
INFO: ../kmeans.c(130): km iter [17] 3.428229e-03 ...
INFO: ../kmeans.c(130): km iter [18] 3.172649e-03 ...
INFO: ../kmeans.c(130): km iter [19] 2.947088e-03 ...
INFO: ../kmeans.c(130): km iter [20] 2.603584e-03 ...
INFO: ../kmeans.c(130): km iter [21] 2.363332e-03 ...
INFO: ../kmeans.c(130): km iter [22] 2.202385e-03 ...
INFO: ../kmeans.c(130): km iter [23] 1.996926e-03 ...
INFO: ../kmeans.c(130): km iter [24] 1.828798e-03 ...
INFO: ../kmeans.c(130): km iter [25] 1.665084e-03 ...
INFO: ../kmeans.c(130): km iter [26] 1.508150e-03 ...
INFO: ../kmeans.c(130): km iter [27] 1.367824e-03 ...
INFO: ../kmeans.c(130): km iter [28] 1.258108e-03 ...
INFO: ../kmeans.c(130): km iter [29] 1.151303e-03 ...
INFO: ../kmeans.c(130): km iter [30] 1.046084e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 31 sqerr 6.340286e+05 conv_ratio 9.343168e-04
INFO: ../main.c(561): 	best-so-far sqerr = 6.340286e+05
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(516): Initializing means using random k-means
INFO: ../main.c(519): Trial 0: 256 means
INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
INFO: ../kmeans.c(130): km iter [1] 1.379302e-01 ...
INFO: ../kmeans.c(130): km iter [2] 3.473147e-02 ...
INFO: ../kmeans.c(130): km iter [3] 1.589796e-02 ...
INFO: ../kmeans.c(130): km iter [4] 9.246683e-03 ...
INFO: ../kmeans.c(130): km iter [5] 6.071259e-03 ...
INFO: ../kmeans.c(130): km iter [6] 4.333799e-03 ...
INFO: ../kmeans.c(130): km iter [7] 3.330068e-03 ...
INFO: ../kmeans.c(130): km iter [8] 2.629796e-03 ...
INFO: ../kmeans.c(130): km iter [9] 2.166465e-03 ...
INFO: ../kmeans.c(130): km iter [10] 1.799240e-03 ...
INFO: ../kmeans.c(130): km iter [11] 1.517641e-03 ...
INFO: ../kmeans.c(130): km iter [12] 1.314749e-03 ...
INFO: ../kmeans.c(130): km iter [13] 1.144948e-03 ...
INFO: ../kmeans.c(143): kmtrineq n_iter 14 sqerr 4.383058e+05 conv_ratio 9.815118e-04
INFO: ../main.c(561): 	best-so-far sqerr = 4.383058e+05
INFO: ../main.c(843): Initializing variances
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(211): alloc'ing 112Mb obs buf
INFO: ../main.c(211): alloc'ing 14Mb obs buf
INFO: ../main.c(211): alloc'ing 56Mb obs buf
INFO: ../main.c(1323): sqerr [0] == 2.574076e+06
INFO: ../main.c(1327): sqerr = 2.574076e+06 tot 1.604393e+03 rms
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/means [1x4x256 array]
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/variances [1x4x256 array]
INFO: ../main.c(1371): No mixing weight file given; none written
INFO: ../main.c(1525): TOTALS: km 0.026x 1.043e+00 var 0.000x 3.315e+00 em 0.000x 0.000e+00 all 0.026x 1.077e+00

Warnings

  1. Message
    INFO: ../main.c(214): alloc'ing 241Mb obs buf
    INFO: ../main.c(518): Initializing means using random k-means
    INFO: ../main.c(521): Trial 0: 256 means
    INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
    INFO: ../kmeans.c(130): km iter [1] 2.012837e-02 ...
    WARNING: "../kmeans.c", line 409: Empty cluster 90
    INFO: ../main.c(554):   -> Aborting k-means, bad initialization
    INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
    INFO: ../kmeans.c(130): km iter [1] 7.065624e-02 ...
    INFO: ../kmeans.c(130): km iter [2] 8.197116e-01 ...
    INFO: ../kmeans.c(143): kmtrineq n_iter 3 sqerr 9.556706e+07 conv_ratio
    4.972178e-05
    INFO: ../main.c(563):   best-so-far sqerr = 9.556706e+07
    INFO: ../main.c(214): alloc'ing 30Mb obs buf
    INFO: ../main.c(518): Initializing means using random k-means
    INFO: ../main.c(521): Trial 0: 256 means
    INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
    INFO: ../kmeans.c(130): km iter [1] 5.450990e-03 ...
    WARNING: "../kmeans.c", line 409: Empty cluster 20
    WARNING: "../kmeans.c", line 409: Empty cluster 42
    WARNING: "../kmeans.c", line 409: Empty cluster 155
    INFO: ../main.c(554):   -> Aborting k-means, bad initialization
    INFO: ../kmeans.c(130): km iter [0] 1.000000e+00 ...
    INFO: ../kmeans.c(130): km iter [1] 5.045656e-03 ...
    WARNING: "../kmeans.c", line 409: Empty cluster 32
    WARNING: "../kmeans.c", line 409: Empty cluster 241
    INFO: ../main.c(554):   -> Aborting k-means, bad initialization
    

    Explanation

    This may happen if the features are byte-swapped, or if your feature file is very small (only a couple of frames or so), or if your features are bad for some other reason. You may have computed cepstra for the header of the audio files in your corpus too, so you may have garbage vectors in your feature files. Garbage vectors can bias the codebooks badly. Make sure that you do not compute feature vectors corresponding to file-headers. To see the cepstral values use the executable "seecep" provided with the SPHINX package. You can use "cepview" too, but it does not check for byte order and sometimes prints garbage even if the cepstral files are OK. Byte swap can be handled through the flag "-mach_endian" in the SPHINX front-end executable.

    The presence of digital zeros in your waveform files can also cause this problem. If a bunch of sample values are zero, and if any of the terms come out to zero in the computation of the melcep, they get set to a large negative number (the log is skipped in the front-end code). In this case the features corresponding to the frames will look like garbage. When there are many zeros in your samples, the garbage number repeats, and this is a good indication that the problem is due to the presence of digital zeros. To avoid this, it is a good idea to add a one-bit dither to your waveform before computing the features. The SPHINX front-end does this for you through a flag called "-dither". The dither is added to the entire signal (and not just to the regions in which it is zero). However, this should not be a cause for concern because adding dither to the whole signal is known to affect recognition performance *positively* in the SPHINX.

  2. Message
    
    

    Explanation

  3. Message
    
    

    Explanation

Error messages

  1. Message
    
    

    Explanation

Unexpected stops

  1. Stop at
    
    

    Explanation


IS MY TRAINING OK?

I've just run the executable mk_mdef_gen

Structure of the correct log file (the lines in red are explanations)

/usr0/egouvea/SourceForge/tmp/sanity/bin/mk_mdef_gen \
 -phnlstfn /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.phonelist \
 -ocimdef /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef \
 -n_state_pm 5

[Switch]      [Default] [Value]
-phnlstfn               /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.phonelist
-inCImdef                      
-triphnlstfn                   
-inCDmdef                      
-dictfn                        
-fdictfn                       
-lsnfn                         
-n_state_pm   3         5      
-ocountfn                      
-ocimdef                /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef
-oalltphnmdef                  
-ountiedmdef                   
-minocc       1         1      
-maxtriphones 100000    100000 
INFO: main.c(95): Will write CI mdef file /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef
INFO: mk_mdef_gen.c(189): 0 single word triphones in input phone list
INFO: mk_mdef_gen.c(190): 0 word beginning triphones in input phone list
INFO: mk_mdef_gen.c(191): 0 word internal triphones in input phone list
INFO: mk_mdef_gen.c(192): 0 word ending triphones in input phone list
INFO: mk_mdef_gen.c(832): 45 n_base, 0 n_tri
INFO: mk_mdef_gen.c(904): Wrote mdef file /usr0/egouvea/SourceForge/tmp/sanity/model_architecture/sanity.ci.mdef

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    
    

    Explanation

Unexpected stops

  1. Stop at
    
    

    Explanation


IS MY TRAINING OK?

I've just run the executable mk_flat

Structure of the correct log file (the lines in red are explanations)

$SphinxTrain/bin/mk_flat \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.ci.mdef \
 -topo /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.topology \
 -mixwfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/mixture_weights \
 -tmatfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/transition_matrices \
 -nstream 4 \
 -ndensity 256

[Switch]  [Default] [Value]
-moddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.ci.mdef
-mixwfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/mixture_weights
-topo               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.topology
-tmatfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/transition_matrices
-nstream  4         4      
-ndensity 256       256    
../main.c(46): Reading model definition file /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.ci.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 47 total models defined (47 base, 0 tri)
INFO: ../model_def_io.c(564): 282 total states
INFO: ../model_def_io.c(565): 235 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
../main.c(53): 47 models defined
INFO: ../s3tmat_io.c(152): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/transition_matrices [47x5x6 array]
INFO: ../s3mixw_io.c(212): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi_flatinitial/mixture_weights [235x4x256 array]

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    INFO: ../model_def_io.c(568): 6 max state/model
    INFO: ../model_def_io.c(569): 20 min state/model
    ../main.c(53): 47 models defined
    ERROR: "../topo_read.c", line 102: EOF encounted while reading version number in /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.topology!?
    

    Explanation

    The topology file has not been made. Check to see if the script that makes the topology is in its right place, that it is executable, and that it is able to write the topology file in the appropriate place.

Unexpected stops

  1. Stop at
    
    

    Explanation


IS MY TRAINING OK?

I've just run the executable bw

Structure of the correct log file (the lines in red are explanations)

INFO: main.c(176): Compiled on Nov 23 2000 at 12:11:34
/path/bin/bw \
 -moddeffn /path/model_architecture/training.ci.mdef \
 -ts2cbfn .semi. \
 -mixwfn /path/model_parameters/training.ci_semi_flatinitial/mixture
_weights \
 -mwfloor 1e-08 \
 -tmatfn /path/model_parameters/training.ci_semi_flatinitial/transition_matrices \
 -meanfn /path/model_parameters/training.ci_semi_flatinitial/means \
 -varfn /path/model_parameters/training.ci_semi_flatinitial/variances \
 -dictfn /path/etc/training.dic \
 -fdictfn /path/etc/training.filler \
 -ctlfn /path/etc/training.fileids \
 -part 1 \
 -npart 1 \
 -cepdir /path/feat \
 -cepext feat \
 -lsnfn /path/etc/training.transcription \
 -accumdir /path/bwaccumdir/training_buff_1 \
 -varfloor 0.0001 \
 -topn 4 \                                    
 -abeam 1e-90 \
 -bbeam 1e-40 \
 -agc none \
 -cmn current \
 -meanreest yes \
 -varreest yes -2passvar no \
 -tmatreest yes \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -ceplen 13

[Switch]   [Default] [Value]
-moddeffn            /path/model_architecture/training.ci.mdef
-tmatfn              /path/model_parameters/training.ci_semi_flatinitial/transition_matrices
-mixwfn              /path/model_parameters/training.ci_semi_flatinitial/mixture_weights
-meanfn              /path/model_parameters/training.ci_semi_flatinitial/means
-varfn               /path/model_parameters/training.ci_semi_flatinitial/variances
-mwfloor   0.00001   1.000000e-08
-tpfloor   0.0001    1.000000e-04
-varfloor  0.00001   1.000000e-04              
-topn      4         4
-dictfn              /path/etc/training.dic
-fdictfn             /path/etc/training.filler
-ctlfn               /path/etc/training.fileids
-nskip
-runlen    -1        -1
-part                1
-npart               1
-cepext    mfc       feat
-cepdir              /path/feat
-segext    v8_seg    v8_seg
-segdir
-sentdir
-sentext   sent      sent
-lsnfn               /path/etc/training.transcription
-accumdir            /path/bwaccumdir/training_buff_1
-ceplen    13        13
-agc       max       none
-cmn       current   current
-varnorm   no        no
-silcomp   none      none
-abeam     1e-100    1.000000e-90
-bbeam     1e-100    1.000000e-40         
-varreest  yes       yes
-meanreest yes       yes
-mixwreest yes       yes
-tmatreest yes       yes
-spkrxfrm
-mllrmult  no        no
-mllradd   no        no
-ts2cbfn             .semi.
-feat                c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-timing    yes       yes
-viterbi   no        no
-2passvar  no        no
-sildelfn
-cb2mllrfn
-spthresh  0.0       0.000000e+00
-maxuttlen 0         0
-ckptintv
INFO: main.c(193): Reading /path/model_architecture/training.ci.mdef
INFO: model_def_io.c(598): Model definition info:
INFO: model_def_io.c(599): 47 total models defined (47 base, 0 tri)
INFO: model_def_io.c(600): 282 total states
INFO: model_def_io.c(601): 235 total tied states
INFO: model_def_io.c(602): 235 total tied CI states
INFO: model_def_io.c(603): 47 total tied transition matrices
INFO: model_def_io.c(604): 6 max state/model
INFO: model_def_io.c(605): 20 min state/model
INFO: s3mixw_io.c(127): Read /path/model_parameters/training.ci_semi
_flatinitial/mixture_weights [235x4x256 array]
WARNING: "mod_inv.c", line 363: Model inventory n_density not set; setting to 
value in mixw file, 256.
INFO: s3tmat_io.c(126): Read /path/model_parameters/training.ci_semi
_flatinitial/transition_matrices [47x5x6 array]
INFO: mod_inv.c(291): inserting tprob floor 1.000000e-04 and renormalizing
INFO: s3gau_io.c(163): Read /path/model_parameters/training.ci_semi_
flatinitial/means [1x4x256 array]
INFO: s3gau_io.c(163): Read /path/model_parameters/training.ci_semi_
flatinitial/variances [1x4x256 array]
INFO: gauden.c(178): 1 total mgau
INFO: gauden.c(150): 4 feature streams (|0|=12 |1|=24 |2|=3 |3|=12 )
INFO: gauden.c(189): 256 total densities
INFO: gauden.c(97): min_var=1.000000e-04
INFO: gauden.c(167): compute 4 densities/frame
INFO: main.c(289): Will reestimate mixing weights
INFO: main.c(291): Will reestimate means
INFO: main.c(293): Will reestimate variances
INFO: main.c(295): Will NOT reestimate MLLR multiplicative term
INFO: main.c(297): Will NOT reestimate MLLR additive term
INFO: main.c(305): Will reestimate transition matrices
INFO: main.c(320): Reading main lexicon: /path/etc/training.dic
INFO: lexicon.c(242): 2697 entries added from /path/etc/training.dic
INFO: main.c(331): Reading filler lexicon: /path/etc/training.filler
INFO: lexicon.c(242): 12 entries added from /path/etc/training.filler
INFO: corpus.c(1304): Will process all remaining utts starting at 0
INFO: main.c(534): Reestimation: Baum-Welch
column defns
        <seq>
        <id>
        <n_frame_in>
        <n_frame_del>
        <n_state_shmm>
        <avg_states_alpha>
        <avg_states_beta>
        <avg_states_reest>
        <avg_posterior_prune>
        <frame_log_lik>
        <utt_log_lik>
        ... timing info ...                                           

utt> 0 st001a000 1415 0INFO: cvt2triphone.c(210): no multiphones defined, no conversion done
498 295 187 477 2.235357e-41 -1.090004e+01 -1.542355e+04 utt 0.801x 1.026e upd 0.801x 1.025e fwd 0.130x 1.007e bwd 0.669x 1.030e gau 64.676x 0.998e rsts 0.302x 1.041e rstf 0.004x 1.067e rstu 0.001x 0.765e

utt> 1 st001a001 1548 0 468 274 176 450 1.952369e-41 -9.813935e+00 -1.519197e+04 utt 0.764x 1.000e upd 0.763x 1.000e fwd 0.125x 0.998e bwd 0.636x 1.001e gau 57.130x 1.004e rsts 0.300x 1.010e rstf 0.005x 0.638e rstu 0.001x 0.760e

utt> 2 st001a002 2025 0 780 372 199 505 2.557036e-41 -1.047046e+01 -2.120267e+04 utt 0.878x 1.000e upd 0.878x 1.000e fwd 0.156x 1.002e bwd 0.720x 1.000e gau 72.144x 0.998e rsts 0.337x 1.005e rstf 0.004x 0.882e rstu 0.001x 0.769e

utt> 3 st001a003 757 0 234 198 141 362 1.814709e-41 -7.576153e+00 -5.735148e+03 utt 0.583x 0.999e upd 0.581x 1.001e fwd 0.091x 1.000e bwd 0.489x 1.000e gau 36.585x 1.016e rsts 0.210x 1.107e rstf 0.003x 1.208e rstu 0.001x 1.060e

utt> 4 st001a004 615 0 132 122 93 240 8.520910e-42 -1.349750e+01 -8.300961e+03 utt 0.398x 1.000e upd 0.398x 0.999e fwd 0.072x 0.986e bwd 0.325x 1.002e gau 17.075x 0.979e rsts 0.154x 0.997e rstf -0.000x 0.000e rstu 0.002x 0.647e

***********and so on till the last utt>erance***********

Warnings

  1. Message
    WARNING: "gauden.c", line 1387: (mgau= 0, feat= 0, density= 202) never observed
    WARNING: "gauden.c", line 1387: (mgau= 0, feat= 0, density= 255) never observed
    WARNING: "gauden.c", line 1387: (mgau= 0, feat= 1, density= 9) never observed
    WARNING: "gauden.c", line 1387: (mgau= 0, feat= 2, density= 23) never observed
    WARNING: "gauden.c", line 1387: (mgau= 0, feat= 2, density= 31) never observed
    

    Explanation

    The first line above means that in the codebook for feature stream 0, the codeword 202 was never observed all through the training. Since you've done a VQ with those very feature vectors, the codeword *should* have been observed at least once! This problem arises due to the fact that all vectors in the training set are not used for vector quantization. The agg-seg executable selects every n_th vector from the feature files for vector quantization. This is not a problem when the feature set size is large. However, if this is done for small amounts of training data, some codewords seemingly become unrepresentative of any sample in the training set. This sounds anomalous, but remember that during training, vectors are not considered in isolation. Instead, the baum-welch executable identifies *groups* of vectors to associate with any codeword. That is how not seeing a particular codeword even once becomes possible.

    If you get such warnings too many times, you should reduce the value given to the -stride flag in agg-seg, and redo the vector quantization. The stride variable decides the degree of sampling of the vector space. Every stride_th vector is selected for the VQ. When the value given is 1, every vector is used.

    Smaller amounts of training data result in increased number of warnings. There warnings may appear even if the stride variable is set to 1, but the number will be greatly reduced. The number will further reduce with more and more baum-welch iterations, as segmentations become more and more accurate.

  2. Message
    WARNING: "../accum.c", line 556: The following seno never occur in the input data
            230 231 232 233 234
    
    Explanation

    If you have run the training in parts, the training data in some parts may not have examples of the phone or triphone to which these senone-ids correspond. That is not a problem. A check on the norm logfile at the end of the current bw iteration can tell you if these (or any of these) senones were seen at all in any other part of the training or not. If the norm logfile also reports them as unseen, then it means that your training data did not have any example of the phone or triphone. Check the model-definition file to find out which models these senones correspond to. It may be that you did have examples of the phone or triphone in your training data, but those utterances either got dropped because they couldn't get force-aligned, or they got dropped during the bw iteration itself because of the failure of the backward pass. Check the bw log files for error messages for failure of backward pass or the force-align log files to see why they didn't get force aligned. Very often the problem is just that you didn't have pronunciations for one or more words (they were not present in the dictionary) in those utterances.

    In any case, if the norm file reports these senones as unseen, and continues to do so for many iterations (ie, if the utterance is not recovered as the models get better), then you will end up with zero model parameters for the particular phone/triphone at the end of your training. That can harm recognition performance. If the phone is a filler, then you continue to train and just not use the filler during decoding by dropping it out from the filler dictionary used during decoding. However, if it is a phone/triphone that cannot be removed from the decode dictionaries, or more realistically, *should* have been trained as a separate acoustic entity, then you must retrain with a training dictionary that does not have that phone. If, for example, the phone is "ZH", you can just call it "SH" in the training dictionary and retrain with ZH dropped out from your phonelist. You will of course have to drop it from the decode dictionary too.

  3. Message
    utt> 27 utterancename  306    0WARNING: "../corpus.c", line 1826: LSN utt id, 
    pathname/utterancename, does not match ctl utt id, utterancename
    

    Explanation

    The warnings can be ignored. The warning above means that for the 27th utterance called "utterancename", your control file contains the entry "pathname/utterancename", while the corresponding transcript has only been labeled by "utterancename". So long as the "utterancename" is the same over this warning, there is no problem. If it isn't, then your control and transcript files are misaligned. That needs to be corrected before you train.

  4. Message
    WARNING: "../mod_inv.c", line 308: # of tied states in mdef file, 755 != # of mixing weight sets, 2965, in ../hmms/09.cd_chmm/1d/mixw
    WARNING: "../mod_inv.c", line 328: Model inventory n_density not set; setting to value in mixw file, 1.
    

    Explanation You are either using the wrong model definition file for this part of the training, or the wrong model parameter files. check these out.

  5. Message

    Explanation

Error messages

  1. Message
    INFO: ../model_def_io.c(562): Model definition info:
    INFO: ../model_def_io.c(563): 47 total models defined (47 base, 0 tri)
    INFO: ../model_def_io.c(564): 282 total states
    INFO: ../model_def_io.c(565): 235 total tied states
    INFO: ../model_def_io.c(566): 235 total tied CI states
    INFO: ../model_def_io.c(567): 47 total tied transition matrices
    INFO: ../model_def_io.c(568): 6 max state/model
    INFO: ../model_def_io.c(569): 20 min state/model
    WARN: "../s3io.c", line 225: Unable to open /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//mod
    el_parameters/tongues3.ci_semi_flatinitial/mixture_weights for reading; No such
    file or directory
    FATAL_ERROR: "../main.c", line 930: initialization failed
    Fri Dec  1 18:36:45 EST 2000                                        
    

    Explanation

    This can happen when the flat initialization fails. Flat initialization can fail for many reasons. Look into the logfile for "makeflat" to find the reason why it failed.

  2. Message
    ERROR: "../backward.c", line 706: alpha(1.882464e-10)
    sum of alphas * betas (7.930106e-24) in frame 751
    ERROR: "../baum_welch.c", line 225: 028c020f ignored
    
    or 
    
    utt> 3 0004N004 447 0 84ERROR: "../backward.c", line 380: final state not
    reached ERROR: "../baum_welch.c", line 245: 00/0004N004 ignored
     utt 0.021x 1.088e upd 0.021x 1.000e fwd 0.021x 1.000e bwd 0.000x 0.000e gau 0.008x 1.257e rsts 0.000x 0.000e rstf 0.000x 0.000e rstu 0.000x 0.000e
    
    

    Explanation

    These errors usually occur due to bad initialization and will disappear after a few iterations. More occassionally some transcripts may be bad. They can be ignored if there aren't too many of them.

  3. Message
    s960207001 3090
    WARNING: "../mk_phone_list.c", line 149: Unable to lookup I(2) in the lexicon
    WARNING: "../next_utt_states.c", line 58: Unable to produce CI phones for utt
    ERROR: "../main.c", line 697: Unable to create utterance HMM; skipping...
    
    utt 0.001x 1.333e upd 0.000x 0.000e fwd 0.000x 0.000e bwd 0.000x 0.000e gau 0.000x 0.000e rsts 0.000x 0.000e rstf 0.000x 0.000e rstu 0.000x 0.000e

    Explanation

    This means that the word I(2) is not present in the training dictionary. Check whether your training dict has the same pronunciation markers as the dictionary you used for force-alignment. If this is not so, your training is wrong.

  4. Message
    INFO: ../main.c(757): Normalizing var
     ERROR: "../gauden.c", line 1389: var (mgau=0, feat=2, density=176,
    component=1) < 0
    

    Explanation
    This happens because we use the following formula to estimate variances:
    variance = avg(x2) - [avg(x)]2
    There are a few weighting terms included (the baum-welch "gamma" weights), but they are immaterial to this discussion. The *correct* way to estimate variances is
    variance = avg[(x - avg(x)]2)
    The two formulae are equivalent, of course, but the first one is far more sensitive to arithmetic precision errors in the computer and can result in negative variances. The second formula is too expensive to compute (we need one pass through the data to compute avg(x), and another to compute the variance). So we use the first one in the sphinx and we therefore get the errors of the kind we see above, sometimes.
    The error is not critical (things will continue to work), but may be indicative of other problems, such as bad initialization, or isolated clumps of data with almost identical values (i.e. bad data).
    Another thing that usually points to bad initialization is that you may have mixture-weight counts that are exactly zero (in the case of semi-continuous models) or the gaussians may have zero means and variances (in the case of continuous models) after the first iteration.
    If you are computing semi-continuous models, check to make sure the initial means and variances are OK. Also check to see if all the cepstra files are being read properly.

  5. Message
    While training untied models, Baum-Wlech exits with the following message:
    FATAL_ERROR: "../ckd_alloc.c", line 149: ckd_calloc_3d failed for caller
    at ../mod_inv.c(278) at .. /ckd_alloc.c(150).
    

    Explanation

    Check the untied model-definition file. This can happen if you are training semi-continuous models and trying to train more triphones than your machine's memory can handle. Reduce the number of triphones by setting the flag "maxdesired" to a lower value (number of triphones) when you generate the untied mdef file. Memory shortage is much less likely to happen for continuous models.

  6. Message
    
    

    Explanation

  7. Unexpected stops

    Message

    INFO: ../corpus.c(1206): Will process 1 utts starting at 0
    length 471990272  size 1887961088   /* I added printf to trace feature
    length in areadfloat.c  */
    areadfloat: /tmp/sls-20000131-000-002.mfc: can't alloc data
    column defns
            <seq>
            <id>
            <n_frame_in>
            <n_frame_del>
            <n_state_shmm>
            <avg_states_alpha>
            <avg_states_beta>
            <avg_states_reest>
            <avg_posterior_prune>
            <frame_log_lik>
            <utt_log_lik>
            ... timing info ...
    utt>     0      sls-20000131-000-002ERROR: "../corpus.c", line 1477:
    MFCC read failed.  Retrying after sleep...
    

    Explanation The feature files are byte-swapped.

    Message

    
    

    Explanation

Back to index


IS MY TRAINING OK?

I've just run the executable norm

Structure of the correct log file (the lines in red are explanations)

$SphinxTrain/bin/norm \
 -accumdir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_3 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_4 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_5 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_6 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_7 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_8 \
 -mixwfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/mixture_weights \
 -tmatfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/transition_matrices \
 -meanfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/means \
 -varfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/variances \
 -feat c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/ \
 -ceplen 13

[Switch]   [Default] [Value]
-accumdir            /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_3 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_4 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_5 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_6 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_7 /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_8
-oaccumdir                  
-tmatfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/transition_matrices
-mixwfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/mixture_weights
-meanfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/means
-varfn               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/variances
-regmatfn                   
-dcountfn                   
-inmixwfn                   
-inmeanfn                   
-invarfn                    
-feat                c/1..L-1/,d/1..L-1/,c/0/d/0/dd/0/,dd/1..L-1/
-ceplen              13     
INFO: ../feat.c(191): Using features c/1..L-1/,d2/1..L-1/d4/1..L-1/,c/0/d2/0/d1(d2/0/),d1(d2/1..L-1/)
INFO: ../main.c(499): Reading and accumulating counts from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1
INFO: ../s3mixw_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/mixw_counts [235x4x256 array]
INFO: ../s3tmat_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/tmat_counts [47x5x6 array]
INFO: ../s3gau_io.c(343): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_1/gauden_counts with means with vars (2pass) [1x4x256 vector arrays]

INFO: ../main.c(499): Reading and accumulating counts from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2
INFO: ../s3mixw_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2/mixw_counts [235x4x256 array]
INFO: ../s3tmat_io.c(92): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2/tmat_counts [47x5x6 array]
INFO: ../s3gau_io.c(343): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//bwaccumdir/tongues3_buff_2/gauden_counts with means with vars (2pass) [1x4x256 vector arrays]

*************and so on till all buffers are read**********************

INFO: ../main.c(743): Normalizing mean for n_mgau= 1, n_stream= 4, n_density= 256
INFO: ../main.c(757): Normalizing var
INFO: ../s3mixw_io.c(212): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/mixture_weights [235x4x256 array]
INFO: ../s3tmat_io.c(152): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/transition_matrices [47x5x6 array]
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/means [1x4x256 array]
INFO: ../s3gau_io.c(188): Wrote /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.ci_semi/variances [1x4x256 array]
Sat Dec  2 11:06:57 EST 2000
Current Overall Likelihood Per Frame = 2.76028
Convergence Ratio = 0.047867

Warnings

  1. Message
    WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 210) never observ
    ed
    WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 226) never observ
    ed
    WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 245) never observ
    ed
    WARNING: "../gauden.c", line 1347: (mgau= 0, feat= 2, density= 250) never observ
    ed                                                                             
    

    Explanation

  2. Message
    The norm log files show a lot senones that haev no observations during
       training.
    

    Explanation

    This can only occur if all the utterances covering these senones die during the training. Alternately, you may have used decision trees that were not built using your training data. It is absolutely necessary to build trees using your training data, or if you have to take short cuts, to make sure that all the senones that you are training for are well represented in your training data. Otherwise you will have senones with zero counts and the decoder begins to do strange things. Even if you remove these senones from the mdef file for decoding, it does not ensure that the rest of the models have been trained properly.

Error messages

  1. Message
    
    

    Explanation

Unexpected stops

  1. Stop at
    
    

    Explanation

Checking the model parameters

The norm executable generates four files which constitute the complete set of model parameters. These files contain the

All of these are binary files. After the norm has successfully run through, check each of these files using the printp tool provided with the SPHINX package. This tool prints each of these files in ascii format. In order to know what to look for, it is first necessary to understand the arrangement of model parameters in each of these files. Semi-continuous model parameter files have the following arrangements:

Assuming that you are working with the standard 4-stream feature set where each stream is represented by 256 codewords, here is a list of what you must look for when checking these files:


IS MY TRAINING OK?

I've just run the executable quick_count

Structure of the correct log file (the lines in red are explanations)

Total no. of words = 3423
Num_Phones = 47, Num_Words = 3423
Sil_Index = 0
#. of cxt-indep phones = 47
within-word triphone finished in 0.0166667 secs.
43 phones can begin a word
46 phones can end a word
83 single-phone words
*(*,*)s finished in 0.3 secs
b/e triphones finished in 0.183333 secs
used       2.90 sec

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    
    

    Explanation

Unexpected stops

  1. Stop at
    
    

    Explanation


IS MY TRAINING OK?

I've just run the executable param_cnt

Structure of the correct log file (the lines in red are explanations)

$SphinxTrain/bin/param_cnt \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef \
 -ts2cbfn .semi. \
 -ctlfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.ctl \
 -lsnfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.newtranscripts \
 -dictfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign \
 -fdictfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign \
 -segdir dummy \
 -paramtype phone

[Switch]   [Default] [Value]
-moddeffn            /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
-ts2cbfn             .semi. 
-ctlfn               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.ctl
-part                       
-npart                      
-nskip                      
-runlen                     
-lsnfn               /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//falignout/tongues.1.newtranscripts
-dictfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign
-fdictfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign
-segdir              dummy  
-segext    v8_seg    v8_seg 
-paramtype state     phone  
INFO: ../corpus.c(1270): Will process all remaining utts starting at 0
INFO: ../main.c(87): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 97344 total models defined (47 base, 97297 tri)
INFO: ../model_def_io.c(564): 584064 total states
INFO: ../model_def_io.c(565): 486720 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
INFO: ../main.c(98): Reading .semi.
INFO: ../main.c(132): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign
INFO: ../lexicon.c(207): 3422 entries added from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.dic.falign
INFO: ../main.c(143): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign
INFO: ../lexicon.c(207): 3 entries added from /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//etc/tongues.filler.falign
INFO: ../param_cnt.c(65): Scanning corpus
[1000] 

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    
    

    Explanation

Unexpected stops

  1. Stop at
    
    

    Explanation


IS MY TRAINING OK?

I've just run the executable make_quests

Structure of the correct log file (the lines in red are explanations)

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    -varfn
    {path]/model_parameters/new_fe.ci_continuous/variances \
     -mixwfn
    [path]/model_parameters/new_fe.ci_continuous/mixture_weights \
     -npermute 168 \
     -niter 0 \
     -qstperstt 20 \
    .....
    .....
    .....
    INFO: ../s3gau_io.c(128): Read
    /sphx_train/hub97/training/model_parameters/new_fe.ci_continuous/means
    [153x1x1 array]
    INFO: ../s3gau_io.c(128): Read
    /sphx_train/hub97/training/model_parameters/new_fe.ci_continuous/variances
    [153x1x1 array]
    FATAL_ERROR: "../ckd_alloc.c", line 109: ckd_calloc_2d failed for caller at
    ../main.c(186) at ../ckd_alloc.c(110)
    

    Explanation make_quests searches 2^npermute combinations several times for the optimal clustering of states. For this, it has to store 2^npermute values (for the comparison). So, setting -npermute to anything greater than 8 or 10 makes the program very slow, and anything over 28 will make the program fail. We usually use a value of 8.

  2. Message
    segmentation fault (or very bad linguistic questions after make_quests
    has run through)
    

    Explanation

    Use the printp tool and check to see if any of your CI model parameters are zero. If they are, then probably the phone(s) for which you tried to train an HMM was not seen in the training data. You can't train what you don't have examples for, so remove the phone(s) from your phonelist and retrain the models.

  3. Message
    
    

    Explanation

Unexpected stops

  1. Stop at
    
    

    Explanation


IS MY TRAINING OK?

I've just run the executable bldtree

Structure of the correct log file (the lines in red are explanations)

$SphinxTrain/bin/bldtree \
 -treefn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned/JH-0.dtree \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.untied.mdef \
 -mixwfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights \
 -ts2cbfn .semi. \
 -mwfloor 1e-30 \
 -psetfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions \
 -phone JH \
 -state 0 \
 -stwt 1.0 0.3 0.1 0.01 0.001 \
 -ssplitmin 1 \
 -ssplitmax 5 \
 -ssplitthr 0 \
 -csplitmin 1 \
 -csplitmax 500 \
 -csplitthr 0

[Switch]   [Default] [Value]
-treefn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned/JH-0.dtree
-moddeffn            /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.untied.mdef
-ts2cbfn   .semi.    .semi. 
-meanfn                     
-varfn                      
-varfloor  0.00001   1.000000e-05
-cntthresh 0.00001   1.000000e-05
-mixwfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights
-psetfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
-phone               JH     
-state               0      
-mwfloor   1e-4      1.000000e-30
-stwt                1.0 0.3 0.1 0.01 0.001
-ssplitthr 8e-4      0.000000e+00
-ssplitmin 1         1      
-ssplitmax 5         5      
-csplitthr 8e-4      0.000000e+00
-csplitmin 1         1      
-csplitmax 100       500    
INFO: ../main.c(150): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.untied.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 6859 total models defined (47 base, 6812 tri)
INFO: ../model_def_io.c(564): 41154 total states
INFO: ../model_def_io.c(565): 34295 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
INFO: ../main.c(188): Building trees for [JH AE AX i] through [JH UW IY i]
INFO: ../main.c(216): Covering states |[16500 16694]| == 195
INFO: ../main.c(223): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights
INFO: ../s3mixw_io.c(151): Read /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_parameters/tongues3.cd_semi_untied/mixture_weights [195x4x256 array]
INFO: ../main.c(261): 39 of 39 models have observation count greater than 0.000010
INFO: ../main.c(65): nrm stwt: 0.709 0.213 0.071 0.007 0.001 
INFO: ../main.c(370): 39-class entropy: 9.277157e+03
INFO: ../main.c(493): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
INFO: ../main.c(530): 256 total simple questions (248 phone; 8 word bndry)

This means that you have 248/4 phone groupings in your linguistic questions file. Every grouping serves as 4 questions, since it can be applied to the left context, the right context, and negated and applied to either contexts. You also have 8/2 word boundary questions in your linguistic questions file, each of which can be applied to either context, effectively giving you 8 word boundary questions. INFO: ../main.c(532): 82 Left Only questions, and 82 Right Only questions

Of the 248/4 phone groupings, since you already have context-specific questions in your linguistic questions file (in this case), you have 82 questions that can be applied to the left context, and 82 that can be applied to the right context. They can also of course be negated and applied. INFO: ../dtree.c(1492): Final simple tree

The root node is first split down to ssplitmax number of leaves using the questions (and their simple negations) The leaves are then exhaustively partitioned into two groups. The question corresponding to the best partition is composed of questions corresponding to these two leaves. It is called a complex question. Thus, a simple tree is first built, and that yields a complex question corresponding to the root node of that tree. The procedure is then applied recursively to the leaves resulting from the branching of the node till there are no more leaves to split (each leaf has only one context). The complex questions are written into the tree file. |( QUESTION0 -1 1.514e+04 1.025e+03 7.812e+02

The "|" indicates that a simple tree is being built. The first string after ( is the name of the simple question that resulted inthe best split of the root node. Following that is the context, with -1 standing for the left context, 1 for the right context and 0 for a wordbounday. Here the context is -1. Following that is the likelihood associated with the root node. The next number is the *increase* in likelihood after the root node was split (using that question), and the final number is the count associated with the node (see the technical section of this manual for further explanantions) | ( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02 | ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02 | ( - 2.574e+03 9 0 QUESTION4_1_L -1 2.751e+02 1.714e+02) | ( - 2.214e+03 9 0 QUESTION1 -1 2.001e+02 1.444e+02)) | ( - 1.613e+03 4 0 QUESTION3_8_L -1 1.565e+02 1.033e+02)) | ( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02 | ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02 | ( - 2.526e+03 1 0) | ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01)) | ( - 2.691e+03 9 0 QUESTION0_8_R 1 2.637e+02 1.447e+02)))

The simple tree is now made. The following lines beginning with s> are simply a repetition of the above information s> ( QUESTION0 -1 1.514e+04 1.025e+03 7.812e+02 s> ( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02 s> ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02 s> ( - 2.574e+03 9 0 QUESTION4_1_L -1 2.751e+02 1.714e+02) s> ( - 2.214e+03 9 0 QUESTION1 -1 2.001e+02 1.444e+02)) s> ( - 1.613e+03 4 0 QUESTION3_8_L -1 1.565e+02 1.033e+02)) s> ( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02 s> ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02 s> ( - 2.526e+03 1 0) s> ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01)) s> ( - 2.691e+03 9 0 QUESTION0_8_R 1 2.637e+02 1.447e+02)))INFO: ../dtree.c(1319): Comp split 0

Compound split 0 is completed, the root node will now be split and in the following, one simple tree will be built for each resulting leaf. The string ": Final simple tree" is written before each simple tree is built. After the two simple trees are built, the one that resulted in the higer likelihood increase will be chosen for a further split, and two simple trees will be built for that. The one that is split will contribute to the second compound question and will be the second compund split. INFO: ../dtree.c(1492): Final simple tree |( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02 | ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02 | ( - 2.526e+03 1 0) | ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01)) | ( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02 | ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01 | ( - 1.084e+03 2 0 SILENCE -1 6.989e+01 6.443e+01) | ( - 2.512e+02 3 0 QUESTION0_9_R 1 6.235e+01 2.025e+01)) | ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01 | ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01) | ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01)))) s> ( QUESTION12 -1 6.853e+03 7.827e+02 3.621e+02 s> ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02 s> ( - 2.526e+03 1 0) s> ( - 5.633e+02 7 0 QUESTION0_11_R 1 9.871e+01 3.687e+01)) s> ( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02 s> ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01 s> ( - 1.084e+03 2 0 SILENCE -1 6.989e+01 6.443e+01) s> ( - 2.512e+02 3 0 QUESTION0_9_R 1 6.235e+01 2.025e+01)) s> ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01 s> ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01) s> ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01))))INFO: ../dtree.c(1492): Final simple tree |( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02 | ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02 | ( QUESTION4_1_L -1 2.574e+03 2.751e+02 1.714e+02 | ( - 8.883e+02 6 0 QUESTION2 -1 1.307e+02 6.173e+01) | ( - 1.410e+03 3 0 QUESTION0_8_R 1 1.033e+02 1.097e+02)) | ( QUESTION1 -1 2.214e+03 2.001e+02 1.444e+02 | ( - 7.658e+02 4 0 QUESTION1_8_R 1 9.251e+01 5.624e+01) | ( - 1.248e+03 5 0 QUESTION0_8_R 1 1.364e+02 8.813e+01))) | ( QUESTION3_8_L -1 1.613e+03 1.565e+02 1.033e+02 | ( - 5.607e+02 2 0 QUESTION3_9_L -1 6.001e+01 4.016e+01) | ( - 8.963e+02 2 0 SILENCE 1 7.711e+01 6.314e+01))) s> ( QUESTION0 1 7.261e+03 4.583e+02 4.191e+02 s> ( QUESTION3_11_L -1 5.189e+03 4.012e+02 3.158e+02 s> ( QUESTION4_1_L -1 2.574e+03 2.751e+02 1.714e+02 s> ( - 8.883e+02 6 0 QUESTION2 -1 1.307e+02 6.173e+01) s> ( - 1.410e+03 3 0 QUESTION0_8_R 1 1.033e+02 1.097e+02)) s> ( QUESTION1 -1 2.214e+03 2.001e+02 1.444e+02 s> ( - 7.658e+02 4 0 QUESTION1_8_R 1 9.251e+01 5.624e+01) s> ( - 1.248e+03 5 0 QUESTION0_8_R 1 1.364e+02 8.813e+01))) s> ( QUESTION3_8_L -1 1.613e+03 1.565e+02 1.033e+02 s> ( - 5.607e+02 2 0 QUESTION3_9_L -1 6.001e+01 4.016e+01) s> ( - 8.963e+02 2 0 SILENCE 1 7.711e+01 6.314e+01)))INFO: ../dtree.c(1319): Comp split 1

The second compound split is completed INFO: ../dtree.c(1492): Final simple tree |( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02 | ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01 | ( SILENCE -1 1.084e+03 6.989e+01 6.443e+01 | ( - 9.718e+02 1 0) | ( - 4.220e+01 1 0)) | ( QUESTION0_9_R 1 2.512e+02 6.235e+01 2.025e+01 | ( - 8.426e+01 2 0) | ( - 1.046e+02 1 0))) | ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01 | ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01) | ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01))) s> ( QUESTION0_8_R 1 2.691e+03 2.637e+02 1.447e+02 s> ( QUESTION0_11_R 1 1.471e+03 1.362e+02 8.469e+01 s> ( SILENCE -1 1.084e+03 6.989e+01 6.443e+01 s> ( - 9.718e+02 1 0) s> ( - 4.220e+01 1 0)) s> ( QUESTION0_9_R 1 2.512e+02 6.235e+01 2.025e+01 s> ( - 8.426e+01 2 0) s> ( - 1.046e+02 1 0))) s> ( QUESTION0_18_R 1 9.555e+02 1.208e+02 5.998e+01 s> ( - 1.947e+02 2 0 SILENCE -1 5.473e+01 1.800e+01) s> ( - 6.400e+02 2 0 SILENCE -1 4.076e+01 4.198e+01)))INFO: ../dtree.c(1492): Final simple tree |( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02 | ( - 2.526e+03 1 0) | ( QUESTION0_11_R 1 5.633e+02 9.871e+01 3.687e+01 | ( QUESTION4_0_L -1 2.358e+02 5.519e+01 1.902e+01 | ( - 4.532e+01 2 0 WDBNDRY_B 0 1.694e+01 6.000e+00) | ( - 1.353e+02 1 0)) | ( QUESTION1 1 2.288e+02 4.944e+01 1.786e+01 | ( - 9.277e+01 2 0 WDBNDRY_B 0 2.191e+01 8.992e+00) | ( WDBNDRY_E 0 8.660e+01 2.737e+01 8.863e+00 | ( - 3.073e+01 1 0) | ( - 2.850e+01 1 0))))) s> ( QUESTION3_0_L -1 3.380e+03 2.908e+02 2.175e+02 s> ( - 2.526e+03 1 0) s> ( QUESTION0_11_R 1 5.633e+02 9.871e+01 3.687e+01 s> ( QUESTION4_0_L -1 2.358e+02 5.519e+01 1.902e+01 s> ( - 4.532e+01 2 0 WDBNDRY_B 0 1.694e+01 6.000e+00) s> ( - 1.353e+02 1 0)) s> ( QUESTION1 1 2.288e+02 4.944e+01 1.786e+01 s> ( - 9.277e+01 2 0 WDBNDRY_B 0 2.191e+01 8.992e+00) s> ( WDBNDRY_E 0 8.660e+01 2.737e+01 8.863e+00 s> ( - 3.073e+01 1 0) s> ( - 2.850e+01 1 0))))) INFO: ../dtree.c(1319): Comp split 2

The third compound split is completed. For an explanation of this algorithm, click here INFO: ../dtree.c(1492): Final simple tree ************* and so on till cssplitmax compound splits are done******* *********usually cssplitmax is set to a very high number, so the****** **** compound splits stop when there is nothing more to split********* INFO: ../dtree.c(1492): Final simple tree |( WDBNDRY_B 0 9.277e+01 2.191e+01 8.992e+00 | ( - 5.843e+01 1 0) | ( - 1.244e+01 1 0)) s> ( WDBNDRY_B 0 9.277e+01 2.191e+01 8.992e+00 s> ( - 5.843e+01 1 0) s> ( - 1.244e+01 1 0))INFO: ../dtree.c(1319): Comp split 27 INFO: ../dtree.c(1319): Comp split 28 INFO: ../dtree.c(1451): stop. leaf nodes are specific INFO: ../dtree.c(1492): Final simple tree |( WDBNDRY_B 0 1.452e+02 3.611e+01 1.582e+01 | ( - 5.028e+01 1 0) | ( - 5.877e+01 1 0)) s> ( WDBNDRY_B 0 1.452e+02 3.611e+01 1.582e+01 s> ( - 5.028e+01 1 0) s> ( - 5.877e+01 1 0))INFO: ../dtree.c(1319): Comp split 29 INFO: ../dtree.c(1319): Comp split 30 INFO: ../dtree.c(1319): Comp split 31 INFO: ../dtree.c(1319): Comp split 32 INFO: ../dtree.c(1319): Comp split 33 INFO: ../dtree.c(1319): Comp split 34 INFO: ../dtree.c(1319): Comp split 35 INFO: ../dtree.c(1319): Comp split 36 INFO: ../dtree.c(1319): Comp split 37 INFO: ../dtree.c(1322): stop. leaf nodes are specific

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    In the file (experimentname).buidtree.(phone).(state).log:
    FATAL_ERROR: "../ckd_alloc.c", line 55: Calloc failed from ../two_class.c(95)
    

    Explanation
    The cd-untied models are probably untrained or very badly trained. Check the cd-untied training logfiles.

  2. Message
    'WARNING: "../main.c", line 183: No triphones involving +SMACK+
    FATAL_ERROR: "../ckd_alloc.c", line 55: Calloc failed from ../main.c(668)'.
    

    Explanation
    No triphones were trained for the phone +SMACK+. In this case, the phone (+SMACK+) is a filler phone, for which we do not train triphones anyway. We do not build trees for filler phones. So if bldtree generates this error message and does notoutput any tree for a filler phone, it's okay - just ignore this.

  3. Message
    FATAL_ERROR: "../main.c", line 276: Fewer state weights than states'.
    

    Explanation
    If you are building N-state HMMs, then you must supply one weight per state to the flag -stwt for bldtree. For example, the setting for 5-state HMMs could be

    -stwt     1.0 0.3 0.1 0.01 0.001
    
    Make sure that the number of weights are equal to the number of states.

  4. Message
    
    

    Explanation

Unexpected stops

  1. Stop at
    INFO: main.c(103): nrm stwt: 0.709 0.213 0.071 0.007 0.001
    INFO: main.c(408): 104-class entropy: 4.708714e+04
    INFO: main.c(531): Reading: /path/model_architecture/tongues.tree_questions
    INFO: main.c(568): 196 total simple questions (188 phone; 8 word bndry)
    INFO: main.c(570): 0 Left Only questions, and 0 Right Only questions
    INFO: dtree.c(1353): Comp split 0
    INFO: dtree.c(1362): stop. b_n->wt_ent_dec (0.000e+00) <= 0   
    

    Explanation

  2. The structure of a tree file

    n_node 75
    0 1 2 1.024970e+03 7.812383e+02 ((!QUESTION0 -1))
    1 3 4 7.827493e+02 3.621280e+02 ((!QUESTION12 -1))
    3 13 14 2.715968e+02 1.446638e+02 ((QUESTION0_8_R 1 !QUESTION0_11_R 1 !QUESTION0_9_R 1)(!QUESTION0_8_R 1))
    13 19 20 1.588734e+02 7.022660e+01 ((QUESTION0_19_R 1 SILENCE -1)(!QUESTION0_19_R 1 !QUESTION0_8_R 1 !SILENCE -1))
    19 43 44 6.583110e+01 4.498003e+01 ((!SILENCE -1))
    43 - - 3.678522e+01 6.000000e+00
    44 - - 5.835646e+02 3.898003e+01
    20 41 42 7.375260e+01 2.524657e+01 ((!QUESTION0_18_R 1))
    41 67 68 3.048508e+01 1.324949e+01 ((!SILENCE -1))
    67 - - 1.570110e+01 3.000000e+00
    68 - - 1.045684e+02 1.024949e+01
    42 - - 1.031438e+02 1.199708e+01
    14 31 32 9.854553e+01 7.443717e+01 ((!QUESTION0_9_R 1 SILENCE -1))
    31 - - 9.718422e+02 5.943530e+01
    32 55 56 4.939473e+01 1.500187e+01 ((!SILENCE -1))
    55 - - 4.219557e+01 4.999490e+00
    56 - - 8.426309e+01 1.000238e+01
    4 9 10 2.908346e+02 2.174642e+02 ((!QUESTION3_0_L -1))
    9 29 30 9.870528e+01 3.687373e+01 ((!QUESTION0_11_R 1))
    29 53 54 4.943643e+01 1.785568e+01 ((!QUESTION1 1))
    53 69 70 2.736811e+01 8.863489e+00 ((!WDBNDRY_E 0))
    69 - - 2.849812e+01 4.000000e+00
    70 - - 3.072878e+01 4.863489e+00
    54 71 72 2.190642e+01 8.992188e+00 ((!WDBNDRY_B 0))
    71 - - 1.244149e+01 2.000000e+00
    72 - - 5.842520e+01 6.992188e+00
    30 49 50 5.518961e+01 1.901805e+01 ((QUESTION4_0_L -1))
    49 73 74 1.694358e+01 5.999987e+00 ((!WDBNDRY_B 0))
    73 - - 1.270456e+01 3.000002e+00
    74 - - 1.567007e+01 2.999985e+00
    50 - - 1.352641e+02 1.301806e+01
    10 - - 2.525817e+03 1.805905e+02
    2 5 6 4.985232e+02 4.191103e+02 ((QUESTION0 1 QUESTION3_11_L -1))
    5 11 12 2.751202e+02 1.714405e+02 ((QUESTION4_1_L -1))
    11 21 22 1.381941e+02 6.173034e+01 ((QUESTION2 -1 !QUESTION0_8_R 1)(!QUESTION1 -1))
    21 57 58 4.674642e+01 2.281793e+01 ((QUESTION0_8_R 1))
    57 63 64 3.611139e+01 1.581815e+01 ((!WDBNDRY_B 0))
    63 - - 5.877301e+01 8.818151e+00
    64 - - 5.028220e+01 6.999999e+00
    58 - - 3.269149e+01 6.999775e+00
    22 37 38 8.078258e+01 3.891241e+01 ((!QUESTION2 -1))
    37 65 66 3.547732e+01 1.299995e+01 ((!WDBNDRY_B 0))
    65 - - 4.989049e+01 6.000001e+00
    66 - - 5.591455e+01 6.999952e+00
    38 - - 3.034155e+02 2.591245e+01
    12 27 28 1.032965e+02 1.097102e+02 ((!QUESTION0_8_R 1))
    27 51 52 5.351306e+01 2.600765e+01 ((!WDBNDRY_E 0))
    51 - - 1.894188e+02 1.800761e+01
    52 - - 6.225969e+01 8.000047e+00
    28 - - 1.001935e+03 8.370250e+01
    6 7 8 3.608895e+02 2.476698e+02 ((!WDBNDRY_E 0))
    7 15 16 2.078070e+02 1.443672e+02 ((!QUESTION1 -1 QUESTION0_8_R 1)(!QUESTION1 -1 !QUESTION0_8_R 1 !QUESTION6 1))
    15 23 24 1.370261e+02 7.212400e+01 ((QUESTION0_8_R 1 !QUESTION5 -1 WDBNDRY_B 0)(!QUESTION0_8_R 1))
    23 45 46 6.179547e+01 2.710147e+01 ((!QUESTION0_8_R 1))
    45 - - 2.103741e+02 1.905201e+01
    46 - - 5.734571e+01 8.049463e+00
    24 39 40 8.045180e+01 4.502253e+01 ((!WDBNDRY_B 0))
    39 - - 1.483478e+02 1.800076e+01
    40 - - 3.054461e+02 2.702177e+01
    16 25 26 1.264272e+02 7.224318e+01 ((QUESTION0_11_R 1 !QUESTION1_8_R 1 !WDBNDRY_B 0)(!QUESTION0_11_R 1))
    25 33 34 8.898907e+01 4.871862e+01 ((!QUESTION0_11_R 1))
    33 59 60 4.497908e+01 2.671408e+01 ((!WDBNDRY_B 0))
    59 - - 1.028076e+02 1.071100e+01
    60 - - 1.498041e+02 1.600309e+01
    34 - - 2.196730e+02 2.200454e+01
    26 47 48 5.541730e+01 2.352456e+01 ((!WDBNDRY_B 0))
    47 - - 1.090463e+02 1.200061e+01
    48 - - 1.080980e+02 1.152395e+01
    8 17 18 1.678853e+02 1.033026e+02 ((!QUESTION3_8_L -1 SILENCE 1))
    17 - - 7.428963e+02 5.414283e+01
    18 35 36 8.536245e+01 4.915981e+01 ((!QUESTION3_10_L -1))
    35 61 62 4.037777e+01 1.399997e+01 ((!SILENCE 1))
    61 - - 7.629050e+01 8.999999e+00
    62 - - 3.061310e+01 4.999975e+00
    36 - - 4.700490e+02 3.515984e+01
    
    

IS MY TRAINING OK?

I've just run the executable prunetree

Structure of the correct log file (the lines in red are explanations)

$SphinxTrain/bin/prunetree \
 -itreedir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned \
 -nseno 4000 \
 -otreedir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000 \
 -moddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef \
 -psetfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions \
 -minocc 0

[Switch]  [Default] [Value]
-moddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
-psetfn             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
-itreedir           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.unpruned
-otreedir           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000
-nseno              4000   
-minocc   0.0       0.000000e+00
INFO: ../main.c(52): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
INFO: ../model_def_io.c(562): Model definition info:
INFO: ../model_def_io.c(563): 97344 total models defined (47 base, 97297 tri)
INFO: ../model_def_io.c(564): 584064 total states
INFO: ../model_def_io.c(565): 486720 total tied states
INFO: ../model_def_io.c(566): 235 total tied CI states
INFO: ../model_def_io.c(567): 47 total tied transition matrices
INFO: ../model_def_io.c(568): 6 max state/model
INFO: ../model_def_io.c(569): 20 min state/model
INFO: ../main.c(58): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
INFO: ../main.c(188): AA-0	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-1	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-2	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-3	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AA-4	65 [0 < 0.000000e+00]
INFO: ../main.c(188): AE-0	102 [0 < 0.000000e+00]
INFO: ../main.c(188): AE-1	102 [0 < 0.000000e+00]

Starting from the right of the first line above, the number 0.000000e+00 is the value you have given to the flag -minocc in the prunetree executable. This flag allows you to specify the minimum number of counts that must be associated with each leaf of a tree. The count associated with any node or leaf is the estimated number of observations in that node or leaf. In first stage of pruning, the trees are pruned entirely with regard to these counts. The number 0.000000e+00 indicates that the count was set to zero in this particular case. The experession [0 < 0.000000e+00] means that there were zero leaves which had counts less than zero. The number 65 means that the tree, after zero nodes were pruned out, was left with 65 leaves. The string AA-0 indicates that this information is about the decision tree for the 1st state of the phone AA INFO: ../main.c(188): AE-2 102 [0 < 0.000000e+00] INFO: ../main.c(188): AE-3 102 [0 < 0.000000e+00] ********** and so on till the last tree for the last phone ****** INFO: ../main.c(188): ZH-3 2 [0 < 0.000000e+00] INFO: ../main.c(188): ZH-4 2 [0 < 0.000000e+00] INFO: ../main.c(205): Prior to pruning n_seno= 25725 INFO: ../main.c(213): n_twig= 9529 INFO: ../main.c(242): Pruning 21725 nodes

The next stage of pruning begins here. In this stage, trees which have been first pruned with regard to the minimum occurance threshold given, are pruned with regard to entropy INFO: ../main.c(274): Root node extracted (ZH 4) from heap INFO: ../main.c(274): Root node extracted (ZH 3) from heap

The first line means that all leaves of the tree for the 5th state of the phone ZH got pruned out and you now have a tree with a single or "root" node. This implies that there will be a single tied state (or senone) representing the 5th state of all triphones for the phone ZH. You can confirm this by looking at the state-ids in the tied state model definition file which you will make using these pruned trees for CD-tied training INFO: ../main.c(274): Root node extracted (ZH 0) from heap INFO: ../main.c(274): Root node extracted (ZH 2) from heap INFO: ../main.c(274): Root node extracted (ZH 1) from heap INFO: ../main.c(274): Root node extracted (OY 3) from heap INFO: ../main.c(274): Root node extracted (OY 2) from heap INFO: ../main.c(274): Root node extracted (OY 4) from heap INFO: ../main.c(274): Root node extracted (OY 1) from heap INFO: ../main.c(312): AA-0 33 INFO: ../main.c(312): AA-1 31

The first line means that after the entropy-based pruning, the tree for the 1st state of the phone AA was left with 33 leaves. The first state of all triphones for the phone AA, therefore, will be represented by 33 different senone ids ************* and so on till the last state of the last phone ********* INFO: ../main.c(312): ZH-0 1 INFO: ../main.c(312): ZH-1 1 INFO: ../main.c(312): ZH-2 1 INFO: ../main.c(312): ZH-3 1 INFO: ../main.c(312): ZH-4 1

Warnings

  1. Message
    
    

    Explanation

Error messages

  1. Message
    
    

    Explanation

Unexpected stops and obscure indications of a problem

  1. Stop at Here are some entries from a logfile that has been written during a seemingly smooth run of prunetree. There are no warnings or error messages, but something is very wrong..
    
    -otreedir           /path/trees/tongues.6000
    -nseno              6000   
    -minocc   0.0       0.000000e+00
    INFO: main.c(87): Reading: /path/model_architecture/tongues.alltriphones.mdef
    INFO: model_def_io.c(598): Model definition info:
    INFO: model_def_io.c(599): 74796 total models defined (47 base, 74749 tri)
    INFO: model_def_io.c(600): 448776 total states
    INFO: model_def_io.c(601): 373980 total tied states
    INFO: model_def_io.c(602): 235 total tied CI states
    INFO: model_def_io.c(603): 47 total tied transition matrices
    INFO: model_def_io.c(604): 6 max state/model
    INFO: model_def_io.c(605): 20 min state/model
    INFO: main.c(93): Reading: /path/model_architecture/tongues.tree_questions
    INFO: main.c(223): AA-0	1 [0 < 0.000000e+00]
    INFO: main.c(223): AA-1	1 [0 < 0.000000e+00]
    INFO: main.c(223): AA-2	1 [0 < 0.000000e+00]
    INFO: main.c(223): AA-3	1 [0 < 0.000000e+00]
    INFO: main.c(223): AA-4	1 [0 < 0.000000e+00]
    INFO: main.c(223): AE-0	1 [0 < 0.000000e+00]
    INFO: main.c(223): AE-1	1 [0 < 0.000000e+00]
    INFO: main.c(223): AE-2	1 [0 < 0.000000e+00]
    INFO: main.c(223): AE-3	1 [0 < 0.000000e+00]
    INFO: main.c(223): AE-4	1 [0 < 0.000000e+00]
    ******** and so on till *******************************
    
    INFO: main.c(223): Z-3	1 [0 < 0.000000e+00]
    INFO: main.c(223): Z-4	1 [0 < 0.000000e+00]
    INFO: main.c(223): ZH-0	1 [0 < 0.000000e+00]
    INFO: main.c(223): ZH-1	1 [0 < 0.000000e+00]
    INFO: main.c(223): ZH-2	1 [0 < 0.000000e+00]
    INFO: main.c(223): ZH-3	1 [0 < 0.000000e+00]
    INFO: main.c(223): ZH-4	1 [0 < 0.000000e+00]
    INFO: main.c(240): Prior to pruning n_seno= 200
    WARNING: "main.c", line 244: n_seno_wanted= 6000, but only 200 defined by trees
    INFO: main.c(248): n_twig= 0
    INFO: main.c(347): AA-0	1
    INFO: main.c(347): AA-1	1
    INFO: main.c(347): AA-2	1
    INFO: main.c(347): AA-3	1
    INFO: main.c(347): AA-4	1
    INFO: main.c(347): AE-0	1
    INFO: main.c(347): AE-1	1
    INFO: main.c(347): AE-2	1
    INFO: main.c(347): AE-3	1
    ******** and so on till *******************************
    
    INFO: main.c(347): Z-2	1
    INFO: main.c(347): Z-3	1
    INFO: main.c(347): Z-4	1
    INFO: main.c(347): ZH-0	1
    INFO: main.c(347): ZH-1	1
    INFO: main.c(347): ZH-2	1
    INFO: main.c(347): ZH-3	1
    INFO: main.c(347): ZH-4	1
    
    

    Explanation

    In the logfile above, you see that in the first stage of pruning, every single tree had only one leaf (the root node). Obviously then, after the second stage of pruning, there was again only one leaf in every tree. The first set of ones indicates that none of the trees ever branched - and that could be due to many reasons. A tree will not branch if the linguistic questions simply repeat with different names in your linguistic questions file, or are bad in some other bizarre manner. First check your linguistic questions file... If that checks out ok, then look at the mixture weights. Are they all zero? Are they all equal? Does the last norm logfile report any errors or warnings? If it does, see if you can trace back the problem from there. If the linguistic questions are back, the trace can begin from the logfile for make_quests.

  • The structure of a pruned tree file

    n_node 9
    0 1 4 1.024970e+03 7.812383e+02 ((!QUESTION0 -1))
    1 2 3 7.827493e+02 3.621280e+02 ((!QUESTION12 -1))
    2 - - 0.000000e+00 1.446638e+02
    3 - - 0.000000e+00 2.174642e+02
    4 5 6 4.985232e+02 4.191103e+02 ((QUESTION0 1 QUESTION3_11_L -1))
    5 - - 0.000000e+00 1.714405e+02
    6 7 8 3.608895e+02 2.476698e+02 ((!WDBNDRY_E 0))
    7 - - 0.000000e+00 1.443672e+02
    8 - - 0.000000e+00 1.033026e+02
    


    IS MY TRAINING OK?

    I've just run the executable tiestate

    Structure of the correct log file (the lines in red are explanations)

    $SphinxTrain/bin/tiestate \
     -imoddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef \
     -omoddeffn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.4000.mdef \
     -treedir /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000 \
     -psetfn /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
    
    [Switch]   [Default] [Value]
    -imoddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.alltriphones.mdef
    -omoddeffn           /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.4000.mdef
    -treedir             /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//trees/tongues3.4000
    -psetfn              /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
    INFO: ../model_def_io.c(562): Model definition info:
    INFO: ../model_def_io.c(563): 78204 total models defined (47 base, 78157 tri)
    INFO: ../model_def_io.c(564): 469224 total states
    INFO: ../model_def_io.c(565): 391020 total tied states
    INFO: ../model_def_io.c(566): 235 total tied CI states
    INFO: ../model_def_io.c(567): 47 total tied transition matrices
    INFO: ../model_def_io.c(568): 6 max state/model
    INFO: ../model_def_io.c(569): 20 min state/model
    INFO: ../main.c(68): Reading: /net/bert/usr7/archive/alf33/usr2/rsingh/tongues3//model_architecture/tongues3.tree_questions
    INFO: ../main.c(86): AA-0: offset 235
    INFO: ../main.c(86): AA-1: offset 252
    

    The first line says that with respect to the first state of the first CI phone, the first tied state of the first state of the phone AA (which happens to be the first phone here) is 235. For the second state of AA, the offset is 252. This means that ALL the triphones of AA have their first states assigned numerical ids which lie between 235 and 252. INFO: ../main.c(86): AA-2: offset 268 INFO: ../main.c(86): AA-3: offset 283 INFO: ../main.c(86): AA-4: offset 300 INFO: ../main.c(86): AE-0: offset 317 *********** and so on till the last state of the last phone ******** INFO: ../main.c(86): ZH-3: offset 4233 INFO: ../main.c(86): ZH-4: offset 4234 INFO: ../main.c(108): n_seno= 4235

    Warnings

    1. Message
      
      

      Explanation

    Error messages

    1. Message
      
      

      Explanation

    Unexpected stops and obscure indications of a problem

    1. Stop at
      /path/bin/tiestate \
       -imoddeffn /path/model_architecture/tongues.alltriphones.mdef \
       -omoddeffn /path/model_architecture/tongues.6000.mdef \
       -treedir /path/trees/tongues.6000 \
       -psetfn /path/model_architecture/tongues.tree_questions
      
      [Switch]   [Default] [Value]
      -imoddeffn           /path/model_architecture/tongues.alltriphones.mdef
      -omoddeffn           /path/model_architecture/tongues.6000.mdef
      -treedir             /path/trees/tongues.6000
      -psetfn              /path/model_architecture/tongues.tree_questions
      INFO: model_def_io.c(598): Model definition info:
      INFO: model_def_io.c(599): 74796 total models defined (47 base, 74749 tri)
      INFO: model_def_io.c(600): 448776 total states
      INFO: model_def_io.c(601): 373980 total tied states
      INFO: model_def_io.c(602): 235 total tied CI states
      INFO: model_def_io.c(603): 47 total tied transition matrices
      INFO: model_def_io.c(604): 6 max state/model
      INFO: model_def_io.c(605): 20 min state/model
      INFO: main.c(102): Reading: /path/model_architecture/tongues.tree_questions
      INFO: main.c(120): AA-0: offset 235
      INFO: main.c(120): AA-1: offset 236
      
      

      The first line says that with respect to the first state of the first CI phone, the first tied state of the first state of the phone AA (which happens to be the first phone here) is 235. For the second state of AA, the offset is 236. This means that ALL the triphones of AA have their first states mapped to 235. This can happen when the tree for the first state of AA has only one node. For a phone like AA which is very common in the language, this is obviously an error. The tree should have been deeper than one node. You have to search the logfiles of bldtree to get to the source of this error) INFO: main.c(120): AA-2: offset 237 INFO: main.c(120): AA-3: offset 238 INFO: main.c(120): AA-4: offset 239 INFO: main.c(120): AE-0: offset 240 ****** and so on till the last state of the last phone ********** INFO: main.c(120): ZH-3: offset 433 INFO: main.c(120): ZH-4: offset 434 INFO: main.c(142): n_seno= 435 AA AA AA s 0 -> 235 AA AA AA s 1 -> 236 AA AA AA s 2 -> 237

      The mode of compilation turns on some compiler directives which cause the program to print out extended log files. These lines may have been printed either due to that or just due to some explicit print in the program. In any case they are of zero use. These lines are simply indicating the state-ids assigned to each triphone. The print has to be disabled in the opensource version. ) **** and thousands of lines of garbage till ****** ZH UW AX i 4 -> 434

      Explanation


    IS MY DECODING SETUP OK?

    The first thing that you must ensure is that the feature files are being correctly computed. If you are about to decode with existing models, then make sure that

    Here's the next quick checklist: