Today during RAC installation I got the following error while running root.sh on the first node

Setting the permissions on OCR backup directory
Setting up NS directories
PROT-1: Failed to initialize ocrconfig
Failed to upgrade Oracle Cluster Registry configuration

 

Then I checked ocrconfig log file and got the following output:

 

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2012-05-17 15:16:35.490: [ OCRCONF][1521206416]ocrconfig starts...
2012-05-17 15:16:35.492: [ OCRCONF][1521206416]Failure initializing ocr in DEFAULT. REBOOT INSTALL. err :[PROC-32: Cluster Ready Ser
vices on the local node is not running Messaging error [9]]
2012-05-17 15:16:35.492: [ OCROSD][1521206416]utstoragetype: /ocfs/ocrfile is on FS type 1952539503. Not supported.
2012-05-17 15:16:35.492: [ OCROSD][1521206416]utopen:6'': OCR location /ocfs/ocrfile configured is not valid storage type. Return c
ode [37].
2012-05-17 15:16:35.492: [ OCRRAW][1521206416]proprinit: Could not open raw device
2012-05-17 15:16:35.492: [ default][1521206416]a_init:7!: Backend init unsuccessful : [37]
2012-05-17 15:16:35.492: [ OCRCONF][1521206416]Failure in initializing ocr in INSTALL level. error:[PROC-37: Oracle Cluster Registry
 does not support the storage type configured]
2012-05-17 15:16:35.492: [ OCRCONF][1521206416]Exiting [status=failed]...

 

There’re a lot of reasons and solutions for this error on the web, but in my case the problem was in the mount point parameter. When getting this error, the mount point parameter was as follows:

 

/dev/dm-1 on /ocfs type ocfs2 (rw,_netdev,heartbeat=local)

 

Then I added “datavolume” parameter as follows:

/dev/dm-2 on /ocfs2 type ocfs2 (rw,_netdev,datavolume,heartbeat=local)

 

And it worked!

Today, after changing password of the sysman user of the repository database of the Grid Control, I wasn’t able to connect to the Grid control and getting the following output:

503 Service Unavailable

This output means that there’s a problem with Repository database connection. In this case, you should check the OMS log files which reside under $OMS_HOME/sysman/log – emoms.trc

When I checked this file, I got the following error:

2012-05-10 15:50:37,740 [AJPRequestHandler-ApplicationServerThread-9] ERROR conn.ConnectionService verifyRepositoryEx.818 – Invalid Connection Pool. ERROR = ORA-01017: invalid username/password; logon denied

2012-05-10 15:51:08,004 [AJPRequestHandler-ApplicationServerThread-6] ERROR conn.ConnectionService verifyRepositoryEx.818 – Invalid Connection Pool. ERROR = ORA-01017: invalid username/password; logon denied

This means that OMS can’t connect to the repository database, as the password of the SYSMAN username was changed. In order to make OMS use new password, we need change two parameter in emoms.properties file which resides  $OMS_HOME/sysman/config directory:

oracle.sysman.eml.mntr.emdRepPwd=test

oracle.sysman.eml.mntr.emdRepPwdEncrypted=false

After making necessary changes I’ve tried to start OMS using the following command:

emctl start oms

But unfortunately I got the same error again … After working a while on this error, I’ve decided to take down all OMS applications using the following command and start them again:

$OMS_HOME/opmn/bin/opmnctl stopall

$OMS_HOME/opmn/bin/opmnctl startall

 

After all, while checking log file, the error disappeared.

 

If you’ve completely lost emoms.properties file, check the following metalink note to recreate it:

How to Recreate the emoms.properties File for 10g OMS from Scratch? [ID 733401.1]

These days I’m responsible on applying CPU patches on some critical databases. Actually I’m thinking to create a new blog post on this topic

But now I would like to share a nice blog post which can be useful to overcome some patch installation issues you get during the installation

Here’s the link:

http://oracletechtips.com/2011/cpu-patch-error-code-73-or-74/

I liked the latest (7) step which was also helped me to overcome “is not writable” error even the mentioned .so file wasn’t in use by any active process

And my last advise : Don’t forget to backup “Oracle Home” folder and even having the backup of the database itself!

During a Disaster Recovery check, I got an error while restoring SPFILE from autobackup:

RMAN> restore spfile to '/tmp/spfile.ora' from '/rman_backup/control_autobackup_8357269245-20120303-00.bkp;

<em>RMAN-00571: ===========================================================</em>
<em>RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============</em>
<em>RMAN-00571: ===========================================================</em>
<em>RMAN-03002: failure of restore command at 03/03/2011 18:06:11</em>
<em>RMAN-06172: no autobackup found or specified handle is not a valid copy or piece

 

I’ve checked all available solutions that I know, but unable to restore it.

I started to confuse if it’s not a backup of the current database, so I used strings command and checked the database name inside the controlfile

[oracle @db] strings /rman_backup/control_autobackup_8357269245-20120303-00.bkp | grep prod

PROD

 

Prod is the name of my database, so this is the correct autobackup.

I’ve automated the RMAN backup and wrote a shell script which copies newly created backup files to the remote FTP host. I’ve copied the controlfile autobackup  from the FTP host to the test machine for the Disaster Recovery. Suddenly I remembered the mode that’s used to copy the backup files via FTP. It’s NOT binary! I’ve checked the size of controlfile that’s in FTP server with the one which is located at the production database. There were different! So it’s really “not a valid copy or piece” as RMAN states.

I’ve added FTP command – “binary”  to the shell script to switch to the binary mode, run the backup script again and got my backups copied to the FTP server. Then I tried to restore spfile from the backup and of course – succeeded.

So if copy RMAN backups to the FTP server, make sure you’re using binary mode and don’t forget to perform a Disaster Recovery to make sure your backups are valid

Kai Yu is a senior engineer and solution architect in Dell Oracle Solutions Engineering Lab  .He has worked with Oracle Technology as an Oracle DBA, Oracle Apps DBA since 1995 and is specializing in Oracle RAC, Oracle E-Business Suite and Oracle VM.  Kai has published 16 technical whitepapers  and given more than 40 technical presentations at major oracle technology conferences worldwide including Oracle OpenWorld, UKOUG, Collaborates, OTN Latin America/APAC Tours, etc. Kai is an Oracle ACE Director and was selected for Oracle ACE spotlight in June 2011, and also was featured  in Oracle Magazine Sept/Oct 2010 issue. Kai has been the president and is the current board member of IOUG Oracle RAC SIG and co-founder and board member of  IOUG Virtualization SIG.  Kai was the winner of  the 2011 OAUG Innovator of Year award . Kai has been active in sharing his Oracle knowledge on his Oracle blog http://kyuoracleblog.wordpress.com/.

Could u please provide answer to the following questions as follows:

• Brief information about yourself and your family

I work for Dell Oracle Solutions Engineering lab. I was originally from China. I live in Austin, Texas, USA with my wife Jin who also works in software industry, and my daughter Jessica who currently is a freshman in high school.

 

• Your education:

I got my BS and MS degree in Computer science in China before I came to US. Then I got my second MS in Computer Science in US.

 

• Your experience with Oracle. When you started first? Has it been interest of your side or just a coincidence?

        My interest with database started with my Mater degree thesis research on relational database more    than 20 years ago. My Oracle database career started in 1995 shortly after when I took a database engineer position in a software company working on OCI and C++, PL/SQL and DBA. On that job, I really enjoyed learning and working on Oracle database technology on both applications development side as well as the database administration side. The technology really fascinated me and I decided to pursue my career in Oracle database.

• What was the motive behind to prefer Oracle? Who you have been influenced by?

  I was motivated by the great flexibility and power of the Oracle database and the strong technology behind it.  In my career with Oracle Technology, I have been really inspired by several people. My first mentor was the chief database architect Chip Young in my early career who taught me how to build a high performance application on Oracle Database. In 1997 I learned the Oracle Parallel server (OPS) technology from an Oracle parallel server architect from Oracle and started working  with Oracle cluster database technology. Since 2006, Nadia Bendjedou and Steven Chan and Erik Peterson at Oracle really inspired me to take my Oracle Technology career to the next level by contributing my efforts to the Oracle community (user groups) and Oracle conferences.  With their great inspiration, in April 2010 I reached a great highlight in my career when I was awarded the Oracle ACE Director rank by Oracle Technology Network

 

• What would your preference of profession if not Oracle?

I would probably end up in software development work.

• What motivates you in your job?

The appreciation and love of the technology always motivate me to learn and try new things in my job.

 

• Do you give lectures on Oracle?

Not official Oracle University class. But I have done several seminars on Oracle RAC SIG, which were recorded and managed by Oracle University and are publically accessible. I also have spoken at several  IOUG web seminars and Oracle technology conferences worldwide such as Oracle OpenWorlds, IOUG/OAUG Collaborate conferences, UKOUG conference, Scotland conference, Ottawa  conference, OTN (Oracle Technology Network) Latin America conference tour and APAC conference tour.

 

• Have you authored any book in Oracle?

I have not finished any book. Recently Syed Jaffar Hussain has been very kind to invite me to co-author his new Oracle RAC book with him

 

• Do you manage with your time as to read books on Oracle? What is the name of the book that you read recently?

By working in an engineering lab, I usually work on very latest technology on which normally there are no many published books around. I spent more time reading Oracle documents, whitepapers, Oracle support notes and Oracle blogs of other colleagues and OTN forms. But I do find a great value to read various published Oracle books to learn the topics in details in a systematic ways.

Here is a list of books I found very useful:

1.        Oracle Database 11g Release 2 High Availability: Maximize Your Availability with Grid Infrastructure, RAC and Data Guard by  Scott Jesse, Bill Burton, Bryan Vongray
2.        Oracle Backup & Recovery: Expert secrets for using RMAN and Data Pump, Kamran Agayev Agamehdi & Aman Sharma
3.        Oracle 11g R1/R2 Real Application Clusters Essentials , by Ben Prusinski, Syed Jaffer Hussain
4.        Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning  by Richmond Shee, Kirtikumar Deshpande and K. Gopalakrishnan
5.        For Oracle Applications DBA, Oracle Applications DBA Field Guild  by  Elke Phelps, Paul Jackson     

• What do you think on OTN forums?

Oracle Technology forums is really a great way for people to share the experience and help each other. A lot of time when I ran into issues, I was able to find some hints or even the answers to my problem. I also try to share my findings on the OTN whenever possible. 

• Do you refer to the documentation? And how often does it happen?

I would say Oracle documentation is my primary resource to learn the new products. Many times I started with the beta version of the product as well as the beta version of the documentations. Last year when I involved  Oracle Enterprise Manager cloud control 12c beta and Oracle VM 3.0 beta, I have heavily relied on the beta documentations and the consultation of  the Oracle product managers, even the engineers from the development team.

 

• What is the biggest mistake you have ever made during your DBA career?

Back to the late 90’s, on the third day after I just started working for a small company, the rented small data center had  some issue with its air conditioning, and I didn’t pay much attention and only forwarded the issue to the data center facility manager to fix it. But the fix was not done immediately. Next day, the database crashed, and I was not able to bring up the database as the database startup always crashed in middle of the instance recovery. After working with Oracle support, the root cause was identified that the data block corruption in the logs caused by the overheated storage pretended the instance recovery from completing. So we had to fix the storage issue and restored the database from the backup and recovered the database to the time right before this corrupted block was written. It was very painful and very time consuming process to get the main revenue generating database back online. Luckily I implemented the online database backup in the very first day when I joined the company.

  

• What was your greatest achievement as an Oracle DBA?

In my 17 years Oracle DBA ad architect career, I have received many awards including three times awards from Dell Executive management, and the 2011 OAUG innovator of year award from OAUG.  But my greatest achievements were that I was able to catch up the latest technology and to be recognized by Oracle technology community with the Oracle ACE director rank and the privilege to get to know many industry leading experts in various areas of Oracle Technology.  

• What is you priority to manage the challenges you face?

My priority is always the technology. My job requires me to research the technology and design and implement the Oracle engineering solutions for various IT organizations.

  

• How would you describe the essence of your success? According to your definition of success, how successful have you been so far?

Hard working, appreciation and interest on technology are the essence of my success. I am pretty happy for what I am doing today.

 

• What are your best skills which make you differ from others?

I got a chance to work on the entire IT stack: server, storage, networking, virtualization, OS,  database, middleware, and applications. This allows me to look at the problems from a broad view

• What’s your major weakness?

I prefer hand on technology work,  not much management type work.

 

• Have you ever lost your spirit? If so, what has been the reason and how have you overcome it?

Not really. But I found hard time working with the people that don’t appreciate technology and don’t appreciate technical skills.

• What is the next success you would like to attain and your efforts to this end?

 I would appreciate if I can get a higher technology ladder like a principle architect position or a technical director position as I have been in my current senior technical position for ten years. But I will continue to focus on my technology interest no matter what.

 

• How do you balance your daily life with your career?

It has been a very challenge to balance my daily life and my career, as I like them both, but there is no enough time. But it may be possible to have both sometime. Last year, I took my daughter with me on my conference trip to England and Scotland, and she found it very interesting and very educational on the trip.

 

• Please describe your one day summary of activities?

I normally get up around 7:30am, taking my daughter to school and get into office around 9:00am. I work in office until 7:00pm. And after dinner, I started working around 8:30pm to 12:30am.  During the night hours, I may also talk to my daughter for her school work or watch some piece of movies or TV on a break.

• How many hours do you work and sleep in a day?

Around 6 hours a day, and depending on the workload or other activities.

• Where and how do you spend your daily, weekly and annual holidays?

I like to spend my holiday in travel around the world, also visiting my parents and brother and sister in China.

• Do you think about Oracle during vacations?

No much, travel is my big favor. I would like to learn the different cultures and places on travel. I also travel to different places in the world for Oracle conferences as well as for vacations.

 

• Do you have time or motivation to go in for any sports? If yes, which kind of sport do you go in for?

I swim and play Ping-Pong (table tennis) and also run sometime.

 

• What’s your favorite meal and non-alcoholic drink?

I don’t have any particular favorite food, but just prefer light taste food, not much non-alcoholic drink either.

• What foreign languages do you know?

Beside of English, I speak standard Chinese, my native language.

• What’s your average typing speed?

I can’t type very fast, as never had any special training on typing

• Have you ever get involved in politics?

No. Not very interested.

• What are your hobbies? 

Swimming, playing Ping-Pong, military/history.

 

• How do you spend your free time?

Being with family, travelling, exercising, reading, watching  movies/TVs, house work etc. 

• What’s your biggest ambition?

Visiting all the countries in the world.

• What would be your advice to the beginners in Oracle?

          Learn the solid foundation of the major; Have a strong passion for the Oracle technology;  be ready for the life time learning.

• Would you like your children to follow in your footsteps or take a different path in life?

That will depend on her interest. I don’t force anything. However, I see technology is a very good career.

• Do you have any followers of you?

I have seen some followers in my blogs. But I found that people are easier to find the materials from my blog entries than from my whitepapers and conferences although my 16 whitepapers and more than 40 conferences presentations listed in my Oracle blog (http://kyuoracleblog.wordpress.com) have much richer materials than those on my blog entries. I plan to find some time to extract some materials from those whitepapers and conference presentations and put them on some the blog entries so that the search engine can pick up them.

 

• What is your vision on the future of Oracle?

I think Oracle is a great company that produces the great Oracle Technology. I love to continue to work with Oracle Technology.

 

• Could you please take a photo in your office near to your desktop?

When importing data from different database, sometimes you get errors like:

ORA-39083: Object type TABLESPACE_QUOTA failed to create with error:
ORA-00959: tablespace 'TBS01' does not exist
Failing sql is:

This means that the tablespace “TBS01″ doesn’t exist in the database where you’re importing the data. For this, you can use REMAP_TABLESPACE option. If you have more than one tablespace that doesn’t exist in the second database, use comma as follows:

REMAP_TABLESPACE=db01_tbs01:db02_tbs,db_01_tbs02:db02_tbs 

 

To get which remap script you need to create, check TABLESPACE_NAME column for the DBA_SEGMENTS view and find in which tablespaces your objects are reside

I strongly believe that you already know that NOT IN doesn’t take NULL values into account and those values are not displayed in the output

Check the following example:

SQL> create table my_table (id number, name varchar2(10));

Table created.

SQL> insert into my_table values(1,'test1');

1 row created.

SQL> insert into my_table values(2,null);

1 row created.

SQL> insert into my_table values(3,'test2');

1 row created.

SQL> commit;

Commit complete.

SQL>
SQL>
SQL> select * from my_table;

ID NAME
---------- ----------
 1 test1
 2
 3 test2

SQL> select * from my_table where name not in ('test2');

ID NAME
---------- ----------
 1 test1

SQL>

 

For more information check the following links:

http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:442029737684

http://jonathanlewis.wordpress.com/2007/02/25/not-in/

There will be an online forum with interesting sessions on BIG Data at February 16th. If you’re interested on Big Data, don’t miss it. Here’s the registration link:

https://event.on24.com/eventRegistration/EventLobbyServlet?target=registration.jsp&eventid=388878&sessionid=1&key=4FEDBC21F62A9834AAE0B109EBBFC918&partnerref=exadata_email1_Jan_Bigdata&sourcepage=register

 

Sometimes we got some folders filled with unknown files either due to any bug where Oracle creates 100 files in a second or due to the OS bugs. And in some cases we’re not able to find in which folder files were created. For that, you can use the simplest command in Linux to get the output for the folders with its size sorted. Here’s the command:

du -s /u01/* | sort -n -r

.......

28G     /u01/oracle/product/10.2.0/db_1/flash_recovery_area

........

 

From the output you will get the folder which consumes more space

I’m getting a lot of emails from readers asking me to share my OpenWorld 2011 presentation files. So I decided to share it with you. Here, you can download the presentation with necessary video tutorials that I’ve shown at OpenWorld 2011. Download the .zip file and extract it under the folder C:\ and make sure you run the presentation from the following folder -c:\OOW 2011 – Kamran Agayev A\

Downloaded 11 times

I’ve also uploaded a part of my speech to youtube, you can watch it online: