I’m I talking to Mr. Raghunadh?
Yes I am.
May I know who this is?
This is pawan from Sun Micro Systems I’m calling for
middleware administrator opening in our company, are you loing for job change?
Yes I’m looking for job change
NP:
Its 15 Days
So how soon you can join we need a person join in one week is it possible for you?
One week or 10 days.
Are you permanent employee for Trigent Software?
I’m contract employee in Trigent Software I’m working for
last 3 and half years.
What is your parent company?
My payroll company is blue soft hyd.
Ctc: 4.5 LPA and I’m expecting 40% Hike.
Ectc: 8 LPA
Can you please tell me about yourself?
Hi this is Raghu from Bangalore I’m working as a Middleware
Administrator in Trigent Software for the last 3 and half years……see the doc!
And I’m created the jdbc data sources and connection pools and
I’ve been following the ideal process,
Which is Incident Management, Change Management and Problem
management?
And my deliverables would be handling the concourse from the
day seen and join in the base calls if there is an issue; we provide support like
DB Activities
There is maintenance activity issue resolved from Deployment
personnel.
Can I ask few Questions?
Yes please.
You said you are working all these right so?
What is the difference between an Alert and
Incident?
Incident happens if there is only one issue/injection for business.
If there is a production incident the tool any application or
failure that is affected in production environment and causing a problem so
that is an incident.
Alert is we set the alerts.
Egg: Disk Space usage, Memory Usage for all that.
There is an alert for Disk Space and Disk Space is affecting your
file system so it should not reach 100%.
So
that’s why we set thresholds.
Disk
Space – 85% = warning
Disk
Space – 95% = critical
When the usage reaches 85 we will get an alert saying that this
file system is consuming please take an action.
Immediately we will login to the server and verify the Disk
Space what is consuming the disk if the logs are full and consuming more space we
can remove the old log or archive log etc.
If there is any data that has not been used for long time or
a temp files some back files which are not required those you can remove with
that we can clear the Disk Space.
For memory alert what is the current consumption of the
memory,
And if it is more than 90% of course it is good it still you
used low memory which is available but at the same time may not exceed whatever
is the limit.
We should find out which process is consuming more memory and
we should reach out to the respected team or application team saying that this application
is consuming more memory please let us know if we have to restart this program
it is memory consumption on the server that is the final case whenever there is
one issue reported then only you have to do this otherwise you should not
restart any application .
So this
is about the memory alerts, disk space alerts and there would be some alerts
which are:
Let say it is for weblogic you will get some stuck threads whenever
it starts 600 miili secs if it reaches more than that then it gives time out
actually. So when you get more no of stuck threads on the server.
Probably you have to restart or it will automatically resolve
by itself through stuck threads.
It continuous to increase the no of stuck threads then you
may get An OutOfMemory also.
Most of the time OutOfMemory is heap space issue or it can be
a Perm Gem space issue or in some cases you may get NativeMemoryError.
Heap Space Issue:-
YOU would recommend the Dev team or App team to increase the
heap size Xmx or Xms values.
That is the resolution for memory consumption whatever it is.
If it is happening every time then we have to check the
oracle support for weblogic.
They will guide you what is causing this issue some
recommendation they would give you have to follow that.
Alert can be Incident why?
If we do not take an action when an alert is come an alert would
be warning or critical whatever it could be .
So when the alert is coming you have to take an action lets
say have not taken any action it is waning and it reaches critical then the
critical alert will be triggered If you don’t take an action even it is
critical alert then it will lead to incident.
Egg: File System Usage is 100%
Then the system hangs it.
At that time you should turn into the incident where you have
to work out along with the Unix Team reboot the server and you have to clear
the space .
So this difference between an incident and alert.
Incident means there is an issue and it is causing against
the business .
If it is a critical incident or major incident that means business
is affected very badly there would be a financial loss or there could be customer
satisfaction during that particular time so that time it would be a very high
incidents or a critical incident .
And for any critical incident or any high incident the bridge
call will be opened everybody has to join there.
What is everybody?
First of all they will call the System Admins that means
system is maintained or middleware team ,
So middleware team will check what is the server, what is the
application, where it is ending is there any errors in the logs they will check
everything .
And if system reports that there is some error in the logs
and it is reporting that the oracle account in the log or there is some ORA
Error then the system may suggests we have seen an error related to the
database for ex it is oracle database.
We have to tell them:-
Please involve the Oracle Database Team
If it is mysql Error you should tell the:-
Please involve the MySQL Database Team.
So then they will join the bridge call and publish the issue.
Or it is network issue you see something relate to network
firewall or network bridge then you will inform the there is network
connectivity issue please involve the network team.
And we have to give stat updates, you know the issue and you
are fixing let’s say you have to restart something tell them I’m going to
restart this particular server , when you are stopping and starting whenever you
are doing something you have to make a notes every time you should give you
dates to the main team or AOC .
What is the change?
Change is nothing but a change to infrastructure or a configuration
file or a set up anything can be a change and all the production changes would
be done along with the cab approval.
What is cab? – Change Advisory Board
Change allegory board so that board that manager has to
approve your change so that we can perform this change in a particular
maintenance window.
Let yours say in some companies they follow the weekly cab.
Every Monday there is a cab.
And everybody will join there we have to present our cab we hold
assign the reports we have to tell the justification also what is the change you
are doing in weblogic we are performing some change from weblogic.
They will ask you why you are doing on production.
No this is done in non-production environment
and what is the successful of art all these things they will ask?
You should provide them a sign of email for the testing team whenever
it is that change we are performing in the lower environment on it is successful
so that we want to fix this in the production environment also please approve.
Then change management will ask if there any other change
along with this on it is going to affect anything.
No one says you don’t have an objection then they will
approve the change.
In some cases let yours say from database change will be
there then the database administrator will tell we have a change for the
foreign client proposal at the same time there might be some deployment plan.
So you have to inform them so we will at least let yours know
when you are going to finish it from then we will start your activity or you
have to postpone your deployment to next week or something.
Because your deployment depends on database you upgrade or
whatever so from change happening on the database side it should not affect you
so it’s always not better to do both changes at same time.
Now you are aware of change management process.
What is problem management?
Problem management is nothing out if there is an incident
from OutOfMemory issue happened and something happened and it keep happening
very frequently so proficient management team will tell that these is reoccurring issue and you should
have a problem payout you for this?
What they will do they will a problem ticket to you and you
have to work on it.
So many questions like: -----
What is the issue, what is the resolution, Is
there a permanent fix, when you will get to you date the permanent fix or if you
don’t have a fix how to proceed for that are you going to work with any vendor
to get this fix or any development team is there who can fix this issue?????
So once the problem ticket is created it will be appended to
the respective team who is going to work on a permanent fix so the problem
ticket will be created only when there is reoccurring issue.
An issue happening very frequently.
Now you were aware of Problem Management, Incident
Management, Change Management and what an alert is also .
How do apply a patch?
I will download the patch for the oracle support related to
the you whatever you suggested once I download it I will place the downloaded bundle
and there will be a directory they will not directly create a cache directory
on their bsu then I will create one then I will place the war file on the xml
file.
You Should have two files on the cache directory:-
One is patch jar file and another one is cat log file (xml
file)
This is the info about the particular patch.
We should remove the readme.txt file from that place.
Once you create these files on the cache directory then we
will you use the bsyou.sh for the patching and will give the patch path and
then be patch list.
And If I want to see how the patching works would I will give
–verbose.
And there is another command once we applied the patch we can
see what are the patches applied on the server.
And also there is a command to remove the patch if a new
patch is released we will remove the existing patch and apply the new patch.
Why can’t you keep the existing patch and applying the new patch?
Oracle always recommends that you cannot do this you have to
remove the old patch and apply the new patch.
This is about patch.
What is the CSR?
CSR is Certificate Signing Request
So on any server which opens ssl command we will generate the csr and key files once you
generate the csr and key files they will provide the csr file to the security
team or to the team who will provide the certificate.
Once it is approved whatever the vendor it is they will give
the certificate along with the Root CA Certificate so now we will have The Root
CA Certificate, CRT certificate and the key file.
We will create these three certificates or the three files on
the Apache we will place these three files in the pack and then we will load it
please keep your renewed certificate file and Root CA Certificate file under
the httpd.conf file in the ssl section then they will mention all these files.
So that is the procedure for certificate renewal.
How do you create the data source?
We create the data sources from the web logic console,,,,,
We will go to the messaging > create generic data source
>new
Then we click on new it will show you few options that is generic
and multi data source we will select the generic data source and we require the
database name, database hostname, database YOUN & PWD.
Along with JNDI name and data source name and which driver you
have to select
If it is XE data source we have to select the jdbc XA.
And if it is non-XA data ource you have to select oracle thin
non-XA .
And there are two types:-
One is Service Connection and instance Connection
We always go with the service connection for better
performance.
So this is how we create the data source.
And we are creating the data source for different database
For oracle we will select the database as oracle and then we
will select the driver.
If it is MySQL database yes you can have the drop down list
in the drop down list you can see the MySql database you should select the
MySql database and then proceed to select the MySql Driver there you can have
the drivers along with that you proceed.
So like this we create the data soyource for any database.
These are the things You shoyould have to create the data soyource.
Db name
Db hostname
Db: - YOUN & PWD
JNDI name
And then targets which you have to create the
data source for XA.
There is a cluster like c1.
You are creating the data source and you are targeting that
data source to a particular cluster.
And when you are creating the data source if you want to
check let yours check the checked connection is successful or not there is an option.
So You can click on the Test connection then it is showing succeeded
msg that means you entered Un & pwd is correct.
If the pwd is wrong it will say the connection is refused or
something invalid/username or password.
Then you have to correct it.
This is about the data source.
How do you deploy the web application in weblogic?
We will place the war file or ear file under the upload
folder of AdminServer from there we will choose the application from the
console and deploy an application on to the respective cluster.
Once we done the installation the application would go to
prepared state for the first time when we are doing the deployment it will go
to the prepared state so we have to start that application it will go to active
state so once that deployment is done you can check the application by going to
deployments tab or setting tab and then click on the link.
That is about the deployment.
The deployment on tomcat server what you do?
You will just place under webapps folder under that you place
this war file and then you restart the tomcat server.
So that application will deploy.
For JBoss you can just go on to the console one you login to
the console you can see the deployments choose the file and place it on the
respective cluster and then activate That’s All.
So In JBoss you have two options:-
Enable and Disable the Application even through the
application is targeted to particular cluster when you disable it goes to
inactive state and when you enable it, it goes to active state.
And the data source creation in weblogic same like web logic.
How do you do the Apache and web logic integration?
We will copy the mod ssl22.so file from the weblogic modules
folder to Apache modules folder and then we will load the module in the httpd.conf
file along with that you have to provide cluster info in the web logic forward
information you have to give the location path if required and if there is any
specific request that have to go to https we have re-writhe the condition on
re-write rule.
So that is how you do the integration.
One is loading the module placing the module in modules
folder and then we have to configure the web logic forward info you will
provide the listen address of web logic server and the port number if you have
a cluster environment you will provide the listen address of particular manage
server and port number,
Listen address of second manages server and port.
So this is about the web logic and Apache integration.
Overview
of Weblogic Server System Administration
Weblogic Server Domain:-
Figure 1-1 depicts a possible configuration of a
WebLogic Server Domain—one of many possible configurations.
Machine A hosts one instance of WebLogic Server, the
Administration Server. The System Administration Tools communicate with the
Administration Server to perform configuration and monitoring of the servers
and applications in the domain. The Administration Server communicates with
each Managed Server on behalf of the System Administration Tools. The
configuration for all the servers in the domain is stored in the configuration
repository, the config.xml file,
which resides on the machine hosting the Administration Server.
Machines B and C each host two
instances of WebLogic Server, WebLogic Servers 1 through 4. These instances are
called Managed Servers. The Administration Server
communicates with an instance of Node Manager running on each machine to
control startup and shutdown of the Managed Servers.
WebLogic Servers 2 and 4 are part of a WebLogic
Cluster (outlined in red). This cluster is running an application
that responds to HTTP requests routed to the cluster from a hardware load
balancer. (An instance of WebLogic Server or a third-party Web server with one
of the WebLogic Server plug-ins can also provide load balancing.) The load
balancer processes HTTP requests from the Internet after they have passed
through a firewall. The load balancer and firewall are not part of the domain.
A replicated copy of objects such as HTTP sessions is passed between the two
cluster members to provide failover capability.
WebLogic Server 1 runs an application
that uses Java Database Connectivity (JDBC) to access a database server running
on another physical machine that is not part of the WebLogic Domain.
Note: The pictured domain is only intended to illustrate the concepts of
a WebLogic Server domain and how you manage the domain. Many possible
configurations of servers, clusters, and applications are possible in a
WebLogic Server domain.
Overview of Node Manager
Node Manager Communications:-
Introduction
Server instances in a WebLogic Server
production environment are often distributed across multiple domains, machines,
and geographic locations. Node Manager is a WebLogic Server utility that
enables you to start, shut down, and restart Administration Server and Managed
Server instances from a remote location. Although Node Manager is optional, it
is recommended if your WebLogic Server environment hosts applications with high
availability requirements.
A Node Manager process is not
associated with a specific WebLogic domain but with a machine. You can use the
same Node Manager process to control server instances in any WebLogic Server
domain, as long as the server instances reside on the same machine as the Node
Manager process. Node Manager must run on each computer that hosts WebLogic
Server instances—whether Administration Server or Managed Server—that you want
to control with Node Manager.
Node Manager
Versions
WebLogic Server provides two versions
of Node Manager, Java-based and script-based, with similar functionality.
However, each version has different configuration and security considerations.
Java-based
Node Manager
Java-based Node Manager runs within a
Java Virtual Machine (JVM) process. It is recommended that you run it as a
Windows service on Windows platforms and as an operating system service on UNIX
platforms, allowing it to restart automatically when the system is rebooted.
Oracle provides native Node Manager
libraries for Windows, Solaris, HP UX, Linux on Intel, Linux on Z-Series, and
AIX operating systems.
Note:
Node Manager
is not supported on Open VMS, OS/390, AS400, UnixWare, or Tru64 UNIX.
This version of Node Manager
determines its configuration from the
nodemanager.properties file. See Reviewing nodemanager.properties.
Java-based Node Manager provides more
security than the script-based version. See Configuring Java-based Node Manager Security.
Script-based
Node Manager
For UNIX and Linux systems, WebLogic
Server provides a script-based version of Node Manager. This script is based on
UNIX shell scripts, but uses SSH for increased security. SSH uses user-id based
security.
For information on configuring the
script version of Node Manager, see Configuring Script Node Manager.
This version does not provide as much
security as the Java-based version. However, the advantage of the script-based
Node Manager is that it can remotely manage servers over a network that has been
configured to use SSH. No additional server installation is required. The
scripts merely have to be copied to the remote machine.
Note:
It is
recommended that you run script-based Node Manager as an operating system
service, which allows it to restart automatically when the system is rebooted.
Determining
Which Node Manager Version to Use
Which version of Node Manager to use
depends on the requirements of your WebLogic Server environment. The following
considerations can help you decide which version is ideal for your environment:
·
If you are installing WebLogic Server
on a Windows system, you must use the Java version of Node Manager. The
scripted version of Node Manager is not supported on Windows.
·
In order to use consensus leasing,
you may see faster performance when using the Java version of Node Manager.
·
The script-based Node Manager
requires a much simpler security configuration than the Java version. RSH and
SSH are generally easier to configure than SSL which is the security method
used by the Java version of Node Manager. The script version of Node Manager
also requires a smaller footprint than the Java version.
·
The Java version of Node Manager can
be used in conjunction with
inetd on
supported UNIX systems. inetd allows
Node Manager to be automatically restarted upon receiving a request on the
configured port.
Accessing
Node Manager
A Node Manager client can be local or
remote to the Node Managers with which it communicates. You access either
version of Node Manager—the Java version or the script-based (SSH) version—from
the following clients: (In addition, an SSH client in the form of a shell
command template is provided for use with the script-based Node Manager.)
·
Administration Server
Administration
Console, from the Environments > Machines > Configuration > Node Manager page.
What You Can
Do with Node Manager
The following sections describe basic
Node Manager functionality.
Start, Shut
Down, and Restart an Administration Server
Using the WebLogic Scripting Tool (or
SSH client for script-based Node Manager only), you connect to the Node Manager
process on the machine that hosts the Administration Server and issue commands
to start, shut down, or restart an Administration Server. The relationship of
an Administration Server to Node Manager varies for different scenarios.
How Node
Manager Starts an Administration Server
Node Manager is running on Machine A,
which hosts the Administration Server. The stand-alone Node Manager client is
remote.
1.
The nm Start command
identifies the domain and server instance to start.
2.
Node Manager looks up the domain
directory in nodemanager.domains, and authenticates the user credentials
using a local file that contains the encrypted username and password.
3.
Node Manager obtains the startup
properties for the Administration Server.
4.
Node Manager creates the
Administration Server process.
5.
The Administration Server obtains the
domain configuration from its
config directory.
Note:
After
the Administration Server is running, you can update the user credentials and
startup properties using the WLST online command,
nmGenBootStartupProps.
How Node
Manager Starts a Managed Server
Node Manager is running on Machine B,
which hosts Managed Server 1. The Administration Server for the domain is
running on Machine A.
1.
From the Administration Console, the
user issues a start command for Managed Server 1.
2. Note:
3. A stand-alone client can also issue a start command for a
Managed Server.
4.
The Administration Server issues a
start command for Managed Server 1 to the Node Manager on the Machine B,
providing the remote start properties configured for Managed Server 1. For
information about the arguments and how to specify them, see Step 5:
Configuring Remote Startup Arguments.
5.
Node Manager starts Managed Server 1.
6.
Node Manager starts the Managed
Server using the same root directory where the Node Manager process is running.
To run the Managed Server in a different directory, set the Root Directory attribute
in the Server > Configuration > Server Start Administration
Console page.
7.
Managed Server 1 contacts the
Administration Server to check for updates to its configuration information.
8.
If there are outstanding changes to
the domain configuration, Managed Server 1 updates its local cache of
configuration data.
How Node
Manager Restarts an Administration Server
Node Manager is running on the machine
that hosts the Administration Server. The Administration Server, which was
initially started with Node Manager, has exited. The Administration
Server's
AutoRestart attribute is
set to true.
Note:
If a server
instance's
AutoRestart attribute
is set to false, Node Manager
will not restart it.
1.
Node Manager determines from the
Administration Server process exit code that it requires restart.
2.
Node Manager obtains the username and
password for starting the Administration Server from the boot.properties file,
and the server startup properties from the server name/data/node manager/startup.
Properties file.
3.
Node Manager starts the
Administration Server.
4.
The Administration Server reads its
configuration data and starts up.
How Node
Manager Restarts a Managed Server
Node Manager is running on Machine B,
which hosts Managed Server 1. Managed Server 1, which was initially started
with Node Manager, has exited. Managed Server 1's
AutoRestart attribute is set to true.
Note:
If a server
instance's
AutoRestart attribute
is set to false, Node Manager
will not restart it.
1. Node Manager determines from Managed Server 1's last known
state that it requires restarting.
2. Node Manager obtains the username and password for
starting Managed Server 1 from the
boot.properties file, and the server startup properties from
the startup.properties file.
These server-specific files are located in the server directory for Managed
Server 1.
3. Node Manager starts Managed Server 1.
Note:
Node Manager waits
RestartDelaySeconds after a server instances fails
before attempting to restart it.
4. Managed Server 1 attempts to contact the Administration
Server to check for updates to its configuration data. If it contacts the
Administration Server and obtains updated configuration data, it updates its
local cache of the
config directory.
5. If Managed Server 1 fails to contact the Administration
Server, and if Managed Server Independence mode (MSI) is enabled, Managed
Server 1 uses its locally cached configuration data.
Note:
Managed Server Independence mode is
enabled by default.
How Node
Manager Shuts Down a Server Instance
Shutting down a Managed Server that
is under Node Manager control.
Node Manager is running on Machine B,
which hosts Managed Server 1.
1. Through the Administration Console, an authorized user
issues a shutdown command for Managed Server 1.
2. The Administration Server issues the shutdown command
directly to Managed Server 1. If it successfully contacts Managed Server 1,
Managed Server 1 performs the shutdown sequence described in "Graceful Shutdown" in Managing Server
Startup and Shutdown for Oracle WebLogic Server.
3. If, in the previous step, the Administration Server failed
to contact Managed Server 1, it issues a shutdown command for Managed Server 1
to Node Manager on Machine B.
4. Node Manager issues a request to the operating system to
kill Managed Server 1.
5. The operating system ends the Managed Server 1 process.
Node Manager
and System Crash Recovery
To ensure that Node Manager properly
restarts servers after a system crash, you must perform the following:
·
For Java-based Node Manager, ensure
that
CrashRecoveryEnabled is
set to true.
The
CrashRecoveryEnabled configuration property allows Node Manager to
restart servers after a system crash. The property is not enabled by default.
·
For script-based Node Manager, place
this line in machine start scripts or, if desired, run periodically on a given
schedule:
· wlscontrol.sh -d domain_name CRASHRECOVERY
·
You should start the Administration
Server using Node Manager.
·
All Managed Servers should be started
using the Administration Server. You can accomplish this using WLST or the
Administration Console.
After the system is restarted, Node
Manager checks each managed domain specified in the
nodemanager.domains file to determine if there are any server instances
that were not cleanly shutdown. This is determined by the presence of any lock
files which are created by Node Manager when a WebLogic Server process is
created. This lock file contains the process identifier for WebLogic Server
startup script. If the lock file exists, but the process ID is not running,
Node Manager will attempt to automatically restart the server.
Note:
When Node
Manager performs a check to access the management servlet, an alert may appear
in the server log regarding improper credentials.
Node Manager
Configuration and Log Files
In managing multiple servers, Node
Manager uses multiple configuration files and outputs log files to multiple
directories, as shown in Figure 2-7.
Figure
2-7 Node Manager Configuration and Logging Environment
The following sections describe Node
Manager configuration and log files:
Configuration
Files
Except where noted, configuration
files apply to both Java-based and script-based Node Manager.
nodemanager.properties
This is the configuration file used
by the Java-based version of Node Manager. See Reviewing nodemanager.properties.
This file is located in WL_HOME
/common/nodemanager, where WL_HOME is the location in which you installed WebLogic
Server.
nodemanager.domains
This file contains mappings between
the names of domains managed by Node Manager and their corresponding
directories. See Step 4: Configuring nodemanager.domains File.
This file is located in WL_HOME
/common/nodemanager.
nm_data.properties
This file stores the encryption data
the Node Manager uses as a symmetric encryption key. The data is stored in
encrypted form.
This file is located in WL_HOME
/common/nodemanager.
nm_password.properties
This file stores the Node Manager
username and password. See Step 2: Specify Node Manager Username and Password.
This file is located in DOMAIN_HOME
/config/nodemanager.
boot.properties
Node Manager uses this file to
specify user credentials when starting a server. See General Node Manager Configuration.
This file is located in DOMAIN_HOME
/servers/server_name/data/nodemanager.
startup.properties
Each Managed Server instance has its
own
startup.properties file
with properties that control how Node Manager starts up and controls the
server. Node Manager automatically creates this file by using properties passed
to Node Manager when the Administration Server was last used to start the
server. This allows a Node Manager client or startup scripts to restart a
Managed Server using the same properties last used by the Administration
Server.
For more information on
startup.properties, see Step 6: Setting Server Startup Properties. These properties correspond to the server startup
attributes contained in ServerStartMBean and the health monitoring attributes
in ServerStartMBean.
This file is located in DOMAIN_HOME
/servers/server_name/data/nodemanager.
Log Files
Use the Node Manager and WebLogic
Server log files to help troubleshoot problems in starting or stopping
individual Managed Servers.
No comments:
Post a Comment