Nowadays, soft skills, such as critical thinking and problem solving are being accentuated among university students as a process of preparing them for job markets. However, the mastery of soft skills is not easily tangible as compared to the hard skills. In this case, an instrument such as a profile is needed to measure the extent of soft skills mastery levels among the students. Thus, a pilot study was conducted to validate and determine the reliability of the self-developed profile. A quantitative method was employed where 119 final year university students who had enrolled in an English course at a Malaysian Technical University Network (MTUN) had participated. The data were analysed based on the Fleiss Kappa and Rasch Model approach. Three experts were consulted for the validity of the instrument (profile). The results showed that all the evaluators gave a very good level of agreement, k=0.95 (>0.80) based on the value of kappa. In addition, the responses from the students were analysed and the Cronbach's Alpha value obtained was high (0.990). In addition, the examination of the functionality of the competency constructed item competency was studied. It was found that respondent reliability was 0.96 (high) and the item reliability value was 0.68 (acceptable). Only two items with the outfit values of MNSQ beyond the range of 0.6 - 1.4 were dropped. Therefore, the study affirmed that the profile was valid and reliable, and could be used to measure the university students’ soft skill abilities in terms of critical thinking skills and problem solving skills. This profile also can be useful for other educational settings and courses.