Welcome Guest Donate | Search | Active Topics | Members | Log In | Register

Building a reliable distributed application on BEA Tuxedo Options
lingy
Posted: Thursday, March 04, 2010 9:44:41 AM

Rank: Administration
Groups: Administration , Member

Joined: 5/8/2009
Posts: 1,578
Points: 5,640

BEA Tuxedo provides the framework, or middleware, for building scalable multi-tier client/server applications in heterogeneous (dissimilar), distributed environments that extend from the Web to the Enterprise. Using BEA Tuxedo, users can develop, manage, and deploy distributed applications independently of the underlying hardware, operating system, network, and database environment.

As indicated in the following figure, middleware consists of software services that exist between a client or server application and the operating system and network services on a system node in the network.

 

Middleware services provide a more functional set of application programming interfaces (API) than the operating system and network services. A main purpose of middleware services is to help solve application connectivity and interoperability problems.

BEA Tuxedo offers the following middleware services:

An ATMI programming interface

ATMI, for Application-to-Transaction Monitor Interface, is the main API for the Tuxedo system. It includes transaction management functions (routines, verbs); request/response, conversational, queuing, and publish-and-subscribe message-handling functions; service interface functions; and buffer management functions for distributed application communication.

A CORBA programming interface

CORBA, for Common Object Request Broker Architecture, is a language-independent, distributed-object model specified by the Object Management Group (OMG). The CORBA programming interface consists of C++ and Java ORBs. An ORB, or Object Request Broker, is a library that enables CORBA objects to locate and communicate with one another.

A high-performance transaction processing application server

The transaction processing application server oversees all aspects of a distributed ATMI transaction, regardless of the systems or resource managers used. It provides the run-time engines for running ATMI transactions on top of ordinary computer hardware and operating systems.

A high-performance object application server

The object application server, based on the CORBA Object Transaction Service (OTS), combines the Tuxedo ATMI transaction processing technology with the BEA CORBA C++ ORB to provide high performance for distributed-object applications using transactions.

BEA Tuxedo includes the ATMI services and CORBA C++ objects needed for transaction management, security, message transport, administration and manageability, and XA-compliant database support for two-phase commit processing. It also includes a high-speed, highly reliable server-side message switch especially tuned for handling distributed transactions across many server machines.

TUXEDO stands for ‘Transactions for Unix Extended for Distributed Operations’.

 

Sponsor
Posted: Thursday, March 04, 2010 9:44:41 AM


lingy
Posted: Thursday, March 04, 2010 10:01:19 AM

Rank: Administration
Groups: Administration , Member

Joined: 5/8/2009
Posts: 1,578
Points: 5,640

The following figure identifies the BEA Tuxedo client and server components and shows the connectivity between the clients and servers. Only remote Tuxedo clients are shown in the figure.

A remote Tuxedo client—ATMI (/WS), Jolt, CORBA C++, CORBA Java, or ActiveX client—interfaces with a Tuxedo server via a network connection and a pair of Tuxedo gateway processes: Workstation Listener/Handler (WSL/WSH), Jolt Server Listener/Handler (JSL/JSH), or IIOP Listener/Handler (ISL/ISH). A remote Tuxedo client may run on a machine that is not part of the Tuxedo server application (typically a workstation or personal computer), or the remote client may run on a machine that is part of the Tuxedo server application. For the latter case, the local operating system intercepts the messages destined for the network and redirects them to the destination process—the Tuxedo remote client or handler process—running locally.

A native Tuxedo client—a native ATMI client or a native CORBA C++ client—is co-located on a machine that is part of the Tuxedo server application and interfaces with a Tuxedo server via the Tuxedo infrastructure using interprocess communication. Native Jolt, CORBA Java, and ActiveX clients are not supported. These clients can only access a Tuxedo server via a pair of JSL/JSH or ISL/ISH gateway processes.

The following brief descriptions of other terms shown in the previous figure should prove helpful in understanding the connectivity between BEA Tuxedo clients and servers:

  • ActiveX: A client-side component model developed by Microsoft. ActiveX is a proprietary technology that runs only on the Windows platform. It is built on top of Component Object Model (COM)/ Distributed COM (DCOM).
  • IIOP: Internet Inter-ORB Protocol. A protocol used for communication between CORBA ORBs over the Internet (TCP/IP).
  • IIOPS: IIOP layered over the SSL protocol.
  • LLE: Link-Level Encryption. A BEA Tuxedo protocol for establishing data privacy over network links between BEA Tuxedo server machines.
  • SSL: Secure Sockets Layer. The standard protocol for establishing secure communications over the Internet (TCP/IP).
BEA Tuxedo Client Components

The following client component software is included in the BEA Tuxedo 8.1 distribution:

  1. BEA ATMI Workstation (/WS) client software
  2. BEA Jolt client software
  3. BEA C++ client ORB including environmental objects
  4. BEA-branded Java client ORB including environmental objects and idltojava compiler
  5. BEA ActiveX client software including environmental objects
  6. 56-bit and 128-bit LLE and SSL encryption software
BEA Tuxedo Server Components

The following server component software is included in the BEA Tuxedo 8.1 distribution:

  1. BEA ATMI server software (includes native ATMI client software)
  2. BEA CORBA C++ server software (includes native CORBA C++ client software)
  3. BEA Jolt server software
  4. BEA SNMP Agent software
  5. BEA Tuxedo Administration Console software
  6. 56-bit and 128-bit LLE and SSL encryption software
lingy
Posted: Thursday, March 04, 2010 10:03:41 AM

Rank: Administration
Groups: Administration , Member

Joined: 5/8/2009
Posts: 1,578
Points: 5,640

The following table lists the invocation capabilities for an application built on the BEA system. A BEA Tuxedo application may span multiple BEA Tuxedo server machines and may provide ATMI services, CORBA objects, or both.

This component . . .

Can call a . . .

Through . . .

ATMI client *

ATMI service

WSL/WSH

Jolt client

ATMI service

JSL/JSH

CORBA C++ client *

CORBA C++ object

ISL/ISH

CORBA Java client

CORBA C++ object

ISL/ISH

ActiveX client

CORBA C++ object

ISL/ISH

ATMI server

ATMI service

Tuxedo infrastructure

CORBA C++ object

CORBA C++ object

Tuxedo infrastructure

CORBA C++ object

ATMI service

Tuxedo infrastructure

* A native Tuxedo ATMI or CORBA C++ client does not use listener or handler gateway processes.

lingy
Posted: Thursday, March 04, 2010 1:23:32 PM

Rank: Administration
Groups: Administration , Member

Joined: 5/8/2009
Posts: 1,578
Points: 5,640

To help you identify error conditions quickly and accurately, the BEA Tuxedo system provides the following log files:

  • Transaction log (TLOG) - binary file that is not normally read by you (the administrator), but that is used by the Transaction Manager Server (TMS). A TLOG is created only on machines involved in BEA Tuxedo global transactions.
  • User log (ULOG) - a log of messages generated by the BEA Tuxedo system while your application is running. The ULOGMILISEC environment variable is used to time stamp ulog message output intervals in milliseconds instead of seconds. The ULOGRTNSIZE environment variable is used to specify rotation files size.

These logs are maintained and updated constantly while your application is running.
 

Transaction Log (TLOG)

The transaction log (TLOG) keeps track of global transactions during the commit phase. At the end of the first phase of a 2-phase commit protocol, the participants in a global transaction issue a reply to the question of whether to commit or roll back the transaction. This reply is recorded in the TLOG.

The TLOG file is used only by the Transaction Manager Server (TMS) that coordinates global transactions. It is not read by the administrator. The location and size of the TLOG are specified by four parameters that you set in the MACHINES section of the UBBCONFIG file.

You must create a TLOG on each machine that participates in global transactions.

The TLOG is a binary file that contains only messages about global transactions that are in the process of being committed. To view the TLOG, you must first convert it to text format so that it is readable. The BEA Tuxedo system provides two tmadmin operations to do this:

  • dumptlog (dl) downloads (or dumps) the TLOG (a binary file) to a text file.
  • loadtlog uploads (or loads) an text version of the TLOG into an existing TLOG (a binary file).

The dumptlog and loadtlog commands are also useful when you need to move the TLOG between machines as part of a server group migration or machine migration.
 

User Log (ULOG)

The user log (ULOG) is a file to which all messages generated by the BEA Tuxedo system—error messages, warning messages, information messages, and debugging messages—are written. Application clients and servers can also write to the user log. A new log is created every day and there can be a different log on each machine. However, a ULOG can be shared by multiple machines when a remote file system is being used.

The ULOG provides an administrator with a record of system events from which the causes of most BEA Tuxedo system and application failures can be determined. You can view the ULOG, a text file, with any text editor. The ULOG also contains messages generated by the tlisten process. The tlisten process provides remote service connections for other machines in an application. Each machine, including the master machine, should have a tlisten process running on it.

The ULOG provides a central repository in which the BEA Tuxedo system and applications can store error information and simplifies the job of finding errors returned by the BEA Tuxedo ATMI.

A ULOG message consists of a tag and text. The tag consists of the following:

  • A 6-digit string (hhmmss) representing the time of day (in terms of hour, minute, and second).
  • The name of the machine.
  • The name and process identifier of the process that is logging the message. (This process ID can optionally include a transaction ID.) Also included is a thread ID (1) and a context ID (0).

The text consists of the following:

  • The message catalog name and number if the log was generated by the BEA Tuxedo system (rather than by the application), such as LIBTUX_CAT:262
  • The literal string gtrid (global transaction identifier) followed by three long hexadecimal integers which uniquely identify the global transaction if the message is sent to the ULOG while the process is in transaction mode, such as gtrid x2 x24e1b803 x239 
  • The message  

The following message are some examples in the ULOG:

162214.mach1!security.23451.1.0: Unknown User 'abc'
162214.mach1!security.23451.1.0: LIBSEC_CAT: 999: Unknown User 'abc'
162214.mach1!security.23451.1.0: gtrid x2 x24e1b803 x239: Unknown User 'abc' 

You can use the information in the ULOG to identify the cause of system or application failures. Multiple messages about a given problem can be placed in the user log. Generally, earlier messages provide more useful diagnostic information than later messages.

In the following example, message 358 from the LIBTUX_CAT catalog identifies the cause of the trouble reported in subsequent messages, namely, that there are not enough UNIX system semaphores to boot the application.

151550.gumby!BBL.28041.1.0: LIBTUX_CAT:262: std main starting 
151550.gumby!BBL.28041.1.0: LIBTUX_CAT:358: reached UNIX limit on semaphore ids
151550.gumby!BBL.28041.1.0: LIBTUX_CAT:248: fatal: system init function ...
151550.gumby!BBL.28040.1.0: CMDTUX_CAT:825: Process BBL at SITE1 failed ...
151550.gumby!BBL.28040.1.0: WARNING: No BBL available on site SITE1.
Will not attempt to boot server processes on that site.

The ULOG is created by the BEA TUXEDO system whenever one of the following events occurs:

  • A new configuration file is loaded
  • An application is booted

When a message is written to the ULOG through the tperrno global variable, application clients and servers are notified, as follows:

  • If tperrno is set to TPESYSTEM after returning from an ATMI call, you can conclude that:
    • A BEA TUXEDO system error has occurred.
    • An error message has been placed in the user log.
  • If tperrno is set to TPEOS after returning from an ATMI call, you can conclude that:
    • An operating system error has occurred.
    • An error message has been placed in the user log. 

BEA Tuxedo system errors indicate problems at the system level, rather than at the application level. When BEA Tuxedo system errors occur, the system writes messages explaining the exact nature of the errors to the central event log (ULOG), and returns TPESYSTEM in tperrno (12). Because these errors occur in the system, rather than in the application, you may need to consult the system administrator to correct them.
 

lingy
Posted: Thursday, March 04, 2010 3:31:54 PM

Rank: Administration
Groups: Administration , Member

Joined: 5/8/2009
Posts: 1,578
Points: 5,640

The Application-to-Transaction Monitor Interface (ATMI), the BEA Tuxedo API, is an interface for communications, transactions, and management of data buffers that works in all environments supported by the BEA Tuxedo system. It provides the connection between application programs and the BEA Tuxedo system. The ATMI is a simple interface for a comprehensive set of capabilities. It implements the X/Open DTP model of transaction processing.

The ATMI library offers you a variety of functions for defining and controlling global transactions in a BEA Tuxedo application. Global transactions enable you to manage exclusive units of work spanning multiple programs and resource managers in your distributed application. All work in a single transaction is treated as a logical unit, so that if any one program cannot complete its task successfully, no work is performed by programs in the transaction. Most ATMI functions support different communication styles. These functions knit together distributed programs by enabling them to send and receive data. All ATMI functions send or receive data in typed buffers. Following is a list of ATMI functions (for C and COBOL bindings), and the tasks they perform. The functions are grouped by task.

Client membership

  • tpchkauth(3c): Check whether authentication is required
  • tpinit(3c): Have a client join an application
  • tpterm(3c): Have a client leave an application

Buffer management

  • tpalloc(3c): Create a message buffer
  • tprealloc(3c): Resize a message buffer
  • tpfree(3c): Free a message buffer
  • tptypes(3c): Get a message type and subtype

Message priority

  • tpgprio(3c): Get the priority of the last request
  • tpsprio(3c): Set the priority of the next request

Request/response communications

  • tpcall(3c): Initiate a synchronous request/response to a service
  • tpacall(3c): Initiate an asynchronous request (fanout)
  • tpgetrply(3c): Receive an asynchronous response
  • tpcancel(3c): Cancel an asynchronous request

Conversational communications

  • tpconnect(3c): Begin a conversation with a service
  • tpdiscon(3c): Abnormally terminate a conversation
  • tpsend(3c): Send a message in a conversation
  • tprecv(3c): Receive a message in a conversation

Reliable queuing

  • tpenqueue(3c): Enqueue a message to a message queue
  • tpdequeue(3c): Dequeue a message from a message queue

Event-based communications

  • tpnotify(3c): Send an unsolicited message to a client
  • tpbroadcast(3c): Send messages to several clients
  • tpsetunsol(3c): Set unsolicited message call-back
  • tpchkunsol(3c): Check the arrival of unsolicited messages
  • tppost(3c): Post an event message
  • tpsubscribe(3c): Subscribe to event messages
  • tpunsubscribe(3c): Unsubscribe to event messages

Transaction management

  • tpbegin(3c): Begin a transaction
  • tpcommit(3c): Commit the current transaction
  • tpabort(3c): Roll back the current transaction
  • tpgetlev(3c): Check whether in transaction mode
  • tpsuspend(3c): Suspend the current transaction
  • tpresume(3c): Resume a transaction

Service entry and return

  • tpsvrinit(3c): Initialize a server
  • tpsvrdone(3c): Terminate a server
  • tpservice(3c): Prototype for a service entry point
  • tpreturn(3c): End a service function
  • tpforward(3c): Forward request

Dynamic advertisement

  • tpadvertise(3c): Advertise a service name
  • tpunadvertise(3c): Unadvertise a service name

Resource management

  • tpopen(3c): Open a resource manager
  • tpclose(3c): Close a resource manager

Note: The use of ATMI transaction management functions is optional.

lingy
Posted: Thursday, March 04, 2010 3:48:29 PM

Rank: Administration
Groups: Administration , Member

Joined: 5/8/2009
Posts: 1,578
Points: 5,640

The BEA Tuxedo system uses the tperrno variable to supply information to a process when a function fails. All ATMI functions that normally return an integer or pointer return -1 or NULL, respectively, on error and set tperrno to a value that describes the nature of the error. When a function does not return to its caller, as in the case of tpreturn() or tpforward(), which are used to terminate a service routine, the only way the system can communicate success or failure is through the variable tperrno in the requester.

The tperrordetail() and tpstrerrordetail() functions can be used to obtain additional detail about an error in the most recent BEA Tuxedo system call on the current thread. tperrordetail() returns an integer (with an associated symbolic name) which is then used as an argument to tpstrerrordetail() to retrieve a pointer to a string that contains the error message. The pointer can then be used as an argument to userlog() or fprintf().

The codes returned in tperrno represent categories of errors, which are listed in the following table.

Error Category

tperrno Values

Abort

TPEABORT2

BEA Tuxedo system1

TPESYSTEM

Call descriptor

TPELIMIT and TPEBADDESC

Conversational

TPEVENT

Duplicate operation

TPEMATCH

General communication

TPESVCFAIL, TPESVCERR, TPEBLOCK, and TPGOTSIG

Heuristic decision

TPEHAZARD2 and TPEHEURISTIC2

Invalid argument1

TPEINVAL

MIB

TPEMIB

No entry

TPENOENT

Operating system1

TPEOS

Permission

TPEPERM

Protocol1

TPEPROTO

Queueing

TPEDIAGNOSTIC

Release compatibility

TPERELEASE

Resource manager

TPERMERR

Timeout

TPETIME

Transaction

TPETRAN2

Typed buffer mismatch

TPEITYPE and TPEOTYPE

1. Applicable to all ATMI functions for which failure is reported by the value returned in tperrno.

2. Refer to Fatal Transaction Errors for more information on this error category.

As footnote 1 shows, four categories of errors are reported by tperrno(5) and are applicable to all ATMI functions. The remaining categories are used only for specific ATMI functions.The following sections describe some error categories in detail.

In the atmi.h, you can find the definitions of the tperrno values:

/*
 * tperrno values - error codes
 * The man pages explain the context in which the following error codes
 * can return.
 */

#define TPMINVAL 0 /* minimum error message */
#define TPEABORT 1
#define TPEBADDESC 2
#define TPEBLOCK 3
#define TPEINVAL 4
#define TPELIMIT 5
#define TPENOENT 6
#define TPEOS  7
#define TPEPERM  8
#define TPEPROTO 9
#define TPESVCERR 10
#define TPESVCFAIL 11
#define TPESYSTEM 12
#define TPETIME  13
#define TPETRAN  14
#define TPGOTSIG 15
#define TPERMERR 16
#define TPEITYPE 17
#define TPEOTYPE 18
#define TPERELEASE 19
#define TPEHAZARD 20
#define TPEHEURISTIC 21
#define TPEEVENT 22
#define TPEMATCH 23
#define TPEDIAGNOSTIC 24
#define TPEMIB  25
#define TPMAXVAL 26 /* maximum error message */

[TPEINVAL]
Invalid arguments were given (for example, svc is NULL or flags are invalid).

[TPENOENT]
Cannot send to svc because it does not exist, or it is a conversational service, or the name provided begins with a dot (.).\

[TPEITYPE]
The type and subtype of idata is not one of the allowed types and subtypes that svc accepts.

[TPEOTYPE]
Either the type and subtype of the reply are not known to the caller; or, TPNOCHANGE was set in flags and the type and subtype of *odata do not match the type and subtype of the reply sent by the service. Neither *odata, its contents, nor *olen is changed. If the service request was made on behalf of the caller's current transaction, then the transaction is marked abort-only since the reply is discarded.

[TPETRAN]
svc belongs to a server that does not support transactions and TPNOTRAN was not set.

[TPETIME]
This error code indicates that either a timeout has occurred or tpcall() has been attempted, in spite of the fact that the current transaction is already marked rollback only. If the caller is in transaction mode, then either the transaction is already rollback only or a transaction timeout has occurred. The transaction is marked abort-only. If the caller is not in transaction mode, a blocking timeout has occurred. (A blocking timeout cannot occur if TPNOBLOCK and/or TPNOTIME is specified.) In either case, no changes are made to *odata, its contents, or *olen. If a transaction timeout has occurred, then, with one exception, any attempts to send new requests or receive outstanding replies will fail with TPETIME until the transaction has been aborted. The exception is a request that does not block, expects no reply, and is not sent on behalf of the caller's transaction (that is, tpacall() with TPNOTRAN, TPNOBLOCK, and TPNOREPLY set). When a service fails inside a transaction, the transaction is put into the TX_ROLLBACK_ONLY state. This state is treated, for most purposes, as though it were equivalent to a timeout. All further ATMI calls for this transaction (with the exception of those issued in the circumstances described in the previous paragraph) will fail with TPETIME.

[TPESVCFAIL]
The service routine sending the caller's reply called tpreturn() with TPFAIL. This is an application-level failure. The contents of the service's reply, if one was sent, is available in the buffer pointed to by *odata. If the service request was made on behalf of the caller's current transaction, then the transaction is marked abort-only. Note that regardless of whether the transaction has timed out, the only valid communications before the transaction is aborted are calls to tpacall() with TPNOREPLY, TPNOTRAN, and TPNOBLOCK set.

[TPESVCERR]
A service routine encountered an error either in tpreturn(3c) or tpforward(3c) (for example, bad arguments were passed). No reply data is returned when this error occurs (that is, neither *odata, its contents, nor *olen is changed). If the service request was made on behalf of the caller's transaction (that is, TPNOTRAN was not set), then the transaction is marked abort-only. Note that regardless of whether the transaction has timed out, the only valid communications before the transaction is aborted are calls to tpacall() with TPNOREPLY, TPNOTRAN, and TPNOBLOCK set. If either SVCTIMEOUT in the UBBCONFIG file or TA_SVCTIMEOUT in the TM_MIB is non-zero, TPESVCERR is returned when a service timeout occurs.

[TPEBLOCK]
A blocking condition was found on the send call and TPNOBLOCK was specified.

[TPGOTSIG]
A signal was received and TPSIGRSTRT was not specified.

[TPEPROTO]
A library routine was called in an improper context. For example, tpcall() was called improperly.

Protocol errors occur when an ATMI function is invoked, either in the wrong order or using an incorrect process. For example, a client may try to begin communicating with a server before joining the application. Or tpcommit() may be called by a transaction participant instead of the initiator.

You can correct a protocol error at the application level by enforcing the rules of order and proper usage of ATMI calls.

To determine the cause of a protocol error, answer the following questions:

  • Is the call being made in the correct order?
  • Is the call being made by the correct process?

[TPESYSTEM]
A BEA Tuxedo system error has occurred. The exact nature of the error is written to a log file.

[TPEOS]
An operating system error has occurred. If a message queue on a remote location is filled, TPEOS may be returned even if tpcall() returned successfully.
 

lingy
Posted: Thursday, March 04, 2010 4:41:41 PM

Rank: Administration
Groups: Administration , Member

Joined: 5/8/2009
Posts: 1,578
Points: 5,640

Determining Types of Failures

The first step in troubleshooting is determining problem areas. In most applications you must consider six possible sources of trouble:

  • Application
  • BEA Tuxedo system
  • Database management software
  • Network
  • Operating system
  • Hardware

Once you have determined the problem area, you must then work with the appropriate administrator to resolve the problem. If, for example, you determine that the trouble is caused by a networking problem, you must work with the network administrator.

How to Determine the Cause of an Application Failure

The following steps will help you detect the source of an application failure.

  1. Check any BEA Tuxedo system warnings and error messages in the user log (ULOG).
  2. Select the messages you think most likely reflect the current problem. Note the catalog name and the number of each of message, so you can look up the message in System Messages. The manual entry provides:
    • Details about the error condition indicated by the message
    • Recommendations for recovery actions
  3. Check any application warnings and error messages in the ULOG.
  4. Check any warnings and errors generated by application servers and clients. Such messages are usually sent to the standard output and standard error files (named, by default stdout and stderr, respectively).
    • The stdout and stderr files are located in the directory defined by the APPDIR variable.
    • The stdout and stderr files for your clients and servers may have been renamed. (You can rename the stdout and stderr files by specifying -e and -o in the appropriate client and server definitions in your configuration file. For details, see servopts(5) in the File Formats, Data Descriptions, MIBs, and System Processes Reference.)
  5. Look for any core dumps in the directory defined by the APPDIR.variable. Use a debugger such as dbx to get a stack trace. If you find core dumps, notify your application developer.
  6. Check your system activity reports (for example, by running the sar(1) command) to determine why your system is not functioning properly. Consider the following reasons:
    • The system may be running out of memory.
    • The kernel might not be tuned correctly.
How to Determine the Cause of a BEA Tuxedo System Failure

The following steps will help you detect the source of a system failure.

  1. Check any BEA Tuxedo system warnings and error messages in the user log (ULOG):
    • TPEOS messages indicate errors in the operating system.
    • TPESYSTEM messages indicate errors in the BEA Tuxedo system.
  2. Select the messages you think most likely reflect the current problem. Note the catalog name and number of each of message, so you can look up the message in System Messages. The manual entry provides:
    • Details about the error condition flagged by the message.
    • Recommendations for recovery actions.
  3. Prepare for debugging in the following ways:
    • Shut down the suspend service.
    • Use tmboot -n -s(server) -d1. (This will not boot the server, but prints the command line used to boot the server by the BEA Tuxedo system.) Use that command line with a debugger such as dbx.
Users browsing this topic
Guest


Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.


© 2010 Canaware Solutions. All rights reserved.
Powered by Canaware Forum version 2.4