Execution
The project documentations were scarce and outdated so I had to rely deeply on the application source code. I identified that there are two distinct purposes for the application. The first is to server as a bridge between two applications called ePrice and FEMS. In this scenario the application is known as Tax Hub (TH). The second purpose is to calculate taxes using the product Taxwere Enterprise from Automatic Data Processing Incorporation. For this purpose the application is known as Taxware Bridge (TB).
There are two instances of TH application running on the Test environment. Each one uses an independent set of queues belonging to two different queue managers. The queue managers are named TAXHUB0 and TAXHUB1. Right after I joined the GTE project I received a request to make the TAXHUB0 point to FEMS Quality Assurance environment. There was no documentation on the queues configuration, so I contacted the MQ Series support group to provide me with a dump of the queue manager's configuration. I identified that it was not possible to solve the problem only by updating the TH configuration files. It was necessary to create new queues on TAXHUB0. I specified the new queues and opened a request on the Support Site application to schedule the queues creation. I updated the TH configuration properties to the new queues. The ePrice team tested the connection and the change was completed successfully.
By the time I joined the project, I received a request to verify why the development environment was not working. Unfortunately there was also no documentation about this environment. So, I started some investigation and I found out that the development environment was as a Windows 2000 server on a company hosting facilities. The server was deactivated because the Windows 2000 Operation System was no longer supported. I contacted the Project Manager and a new AIX server was made available to the project. I received all passwords including the root one. I was responsible for reproducing the entire Windows 2000 environment on the new AIX server. I extracted and recreated the DB2 tables, transferred the table contents. I downloaded a MQ Series dump tool from Internet. Extracted the queue definitions from the Windows 2000 Server. I did some cleanups on this dump. I recreated the queue definitions. I recreated the TH application EAR file from our CVS code repository using Rational Software Architect (RSA) and deployed on the WebSphere Application Server. I fixed some details like the JDBC Datasource configuration to make the application access the database, the listeners on the MQ Series that were not started and some minor other issues. Finally I got the new development environment up and running.
Since I have been responsible for the TH application I have been engaged to help on some Severity 1 problems on production. I had to analyze the TH log files and guide the team on isolating the problem. I was able to identify the following problems on different circumstances:
-
I identified through the logs that some SOAP calls seamed to be stopped and a queue from FEMS to TH started to accumulate messages. The ePrice team informed me that there were three ePrice servers that consume the messages through WebServers. Therefore, there might be some of the three servers that might have some problems. Knowing that, they started to turning a server off and leaving the two other servers up. After a couple of cycles the server with problem was identified. The Severity 1 problem was solved successfully.
-
I identified through the logs that the acknowledge messages were being received from FEMS Brazil but no response was being received. Therefore, there could be a problem on the FEMS application or on the queue from FEMS to TH. The MQ support team identified a channel that was on retry mode. The problem was fixed and we solved the Severity 1 problem.
-
I identified through the logs that the FEMS Brazil was sending neither the taxes nor the acknowledge message back to the TH application. Therefore, there might be a problem on the FEMS application. They FEMS support team identified that there was a file system with no free space on their side and we solved the Severity 1.
The others Severity 1 problems were variations of these three examples. The TH application is critical to the business and needs some constant care because the TH was implemented as a single thread application. Therefore, if any problem happens on any step then the entire flow is compromised.
I was contacted by the DB2 support team and they informed that the two DB2 instances had Fix Pack 16 and had to be upgraded to Fix Pack 17 in order to be in compliance with internal rules. I had to coordinate this change with the customer and open the Support Site ticket. I started by creating a plan and contacting the groups involved. I found out that both the Test and Production environment were based on shared servers. These servers were not only used by the TH but also by another application called WWPRT. I confirmed that this application was also using an instance on the same DB2 installation and had also to be stopped during the change. I contacted the group responsible for the queue NUS_N_SSBRCONN to verify if the fix could be applied on the two servers. They informed that the Test environment did not have enough space to download the Fix Pack. I coordinated with the group responsible for the queue NUS_N_DCRCHAIX to mount a file system to receive the Fix Pack and unmount it after the change. I coordinated with the teams responsible for queues NUS_N_SSUDBDBA and NUS_N_SSWEBNORT to stop the DB2 instances and stop the WebSphere applications respectively. I elaborated a plan estimating 6h to completion and created the tickets. After fixing some small details the tickets got approved. The upgrade was implemented successfully on both servers.
Both test and production environment had the WebSphere Application Server processes running as root user. There was an old request to move them to run on taxeadm user. I had to engage many people from support team to obtain the history of why the exposure had not been eliminated yet. The PM sent a note recognizing my good written communications that overcame the bureaucracy on obtaining the necessary rationales. So, I could develop a sequence of tasks to eliminate the exposure. I created the tickets and coordinated their executions. The problem was solved.
I was responsible for providing assistance on many other tasks like assisting the MQ Series upgrade on both test and production environment, analyze quotes and taxes on many occasions, fix the log configuration for the log4j library to remove duplicated lines, to create "awk" scripts to filter the log and provide friendly information to users, assist the ePrice team on fixing a frequent SOAP exception that got resolved when a wrong library version was replaced with a correct one and on many other assignments similar to these ones.
Right after I joined the GTE project, the CCE team requested the creation of a new environment. Therefore, I had to provide them with all the code and instructions to setup a brand new TB instance. I retrieved the database DDLs, the instructions that I had to install the Taxware Enterprise product and the latest TB code from the code repository CVS. This code produced a different behavior than expected. There was an extra tag in the WSDL definition and the XML files being exchanged in the SOAP calls were not matching the CCE format. I started an investigation and I found that the code repository had only one release and the latest code. Unfortunately, none of the two versions had the proper behavior. There was no documentation about the application releases. Therefore, I concluded that the code was not versioned correctly at the exact moment when the code was delivered to the support team of the CCE application. I had to generate a plan to analyze the parts of the code, fix the extra tag problem, document the findings and do some code cleanups along the way. I generated a plan that includes a resource from India group. We worked as a team, although I coordinated his work to generate a specific test automation tool for TB. I implemented a test version of the TB application that stores on file the strings being exchanged with the Taxware product. I then implemented a tool to break these strings back into a sequence of fields so I could easily analyze the content. Using this technique I could easily verify what was being generated by the TB and what responses the Taxware was providing. This way I was able to safely eliminating the extra tag that was appearing incorrectly. I documented the code and I discovered that the new environment was not correctly initializing some fields. I fixed this problem on the code, I executed a set of tests, and I delivered a final version to the support team of the CCE application. I executed the plan successfully, on time and on budge. During this phase I held many conference calls with CCE team and with the project sponsors to explain the problems identified, to show the action plans I created, to provide updates and to answer questions using a succinct communication.
The updates of taxes on the Taxware Product were not done automatically. I received monthly requests to download taxes update files from ADP site and apply on all TB servers. I uploaded the files on a specific directory and triggered the update using a Motif Graphical Interface that I opened locally by using the Exceed product. This process was time consuming, so I study the Taxware documentation and I was able to create a single script to update the entire product at once.
There were groups that were using a back level of Taxware product compared with the versions installed on the servers used by the TB application. I used the development environment to install these versions. After studying carefully the many steps to install the Taxware product I was able to have many different versions of the Taxware product on a single server and I could easily select which version would be the active one. This way I could provide much better support to other groups.
The TB application applies rules to the quotes. These rules are bases on information stored on a database. I also received requests to update these tables. I had to convert normal Windows Excel files into a DB2 IXF format. Create instructions to DB2 support team, negotiate an application down time with the users, specially the ePrice team because the TB application affects the TH application directly, and finally open a ManageNow ticket to execute the update.
