Abstract: In order to solve the problem of human-computer interaction in virtual assembly systems,driver classes for data glove and 3D tracker are formed,integrated with virtual assembly environment.Methods of assessing and strategies of processing raw data are given.The method of gesture recognition that is synthetically using data collected by the data gloves and 3D trackers are proposed.Modeless dynamic transparent menu and dialog are used to aid human-computer interaction.The virtual assembly process which is realized through operating virtual assembly and part model by virtual hand is discussed.Verification for application of human-computer interaction is committed by developing a 3 channels projection-based virtual assembly environment.The application results manifest that the approach mentioned above is feasible and efficient.